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STUDY  TITLE:  APPLYING  PROGRAM  MANAGEMENT  CONCEPTS  TO  SOFTWARE 
DEVELOPMENT:  AN  AFR  300-15  CRITIQUE 


STUDY  PROJECT  GOALS: 

To  identify  and  propose  changes  to  AFR  300-15  in  an  effort  to 
further  adapt  concepts  and  principles  of  program  management  to 
computer  software  development. 


STUDY  REPORT  ABSTRACT: 

The  USAF  has  recently  incorporated  concepts  and  principles  of 
computer  software  with  preparation  of  AFR  300-15.  However,  it  is 
believed  that  those  concepts  were  not  tailored  finitely  enough. 
Further,  AFR  300-15  appears  to  adapt  the  development  process  to  fit 
the  principles  rather  than  the  reverse. 

This  paper  proposes  changes  to  AFR  300-15  which  tailor/adapt 
the  principles  and  concepts  of  program  management  to  the  software 
developmental  process.  This  is  accomplished  by  realignment  of 
some  functions  within  the  phases,  by  renaming  baselines  and  reviews, 
by  shifting  reviews,  and  by  establishing  some  new  baselines  and  re- 
views. 
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EXECUTIVE  SUMMARY 

1.  Historically,  the  computer  software  development  process  has 

i 

been  difficult  for  managers  to  control.  Systems  design  and  pro-  \ 

gramming  efforts  were  frequently  thought  of  as  highly  individu-  j 

alistic  and  artistic  endeavors.  These  phenomonea  are  doubt-  I 

] 

lessly  attributable  tc  the  relatively  short,  yet  explosive,  life  | 

of  the  Electronic  Data  Processing  industry.  1 

■I 

2.  With  personnel  costs  rising  drastically,  and  with  hardware  j 

costs  rising  at  a lesser  pace,  ways  must  be  found  to  reduce  j 

software  costs  as  well  as  reduce  the  attendant  risks.  I 

-■] 

3.  Industry  and  trade  literature  is  replete  with  "cure-alls". 

However,  the  U.S.  Air  Force  has  chosen  to  approach  the  problem 

by  applying  the  program  management  principles  and  concepts  of  • 

> 

phasing,  baselining,  and  formalized  reviews  and  audits  to  com-  i 

puter  software  development.  This  has  the  property  of  causing  | 

the  developmental  cycle  to  be  broken  into  its  various  component  i 

parts  and  each  attacked  separately,  yet  with  integration  in  j 

mind.  This  approach  is  documented  in  AFR  300-15  Automated  Data  j 

System  Project  Management.  It  is  an  adaptation  of  the  practices 
widely  employed  by  DOD  and  industry  in  weapons  systems  acquisition.  j 

4.  This  paper  is  a critique  of  AFR  300-15.  This  critique  re- 
sults in  specific  proposed  changes  to  AFR  300-15.  This  paper  will 
be  presented  to  Hq.  USAF/KRA  upon  the  author's  graduation  from 
DSMC.  The  changes  include  the  following: 


of  the  document 


In  summary,  these  changes  will  cause  the  document  to  reflect  the 


methodology  shown  in  figure  1 
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Background 

1.  "Today,  providing  computer  software  involves  greater  cost 
and  risk  than  providing  computer  equipment,  because  hardware  is 
mass  produced  by  industry  using  proven  technology,  while  soft- 
ware is  still  produced  mostly  by  the  craft  of  individual  com- 
puter programmers  and  users.  Over  70%  of  all  software  is  de- 
veloped by  the  users  of  computers,  rather  than  the  manufacturers 
of  computer  equipment  or  the  producers  of  commercial  software. 
Software  is  very  expensive  and  the  development  costs  for  new 
systems  are  growing  to  enormous  levels.  The  Federal  government 
is  estimated  to  spend  as  much  as  $5  billion  annually  on  soft- 
ware, and  its  accumulated  investment  in  current  software  proba- 
bly exceeds  $25  billion.  Software  costs  are  projected  to 
become  9 times  as  great  as  hardware  costs  by  1985"  (12) . 

This  quote  is  only  one  among  many  which  could  be  cited 
which  attest  to  the  magnatude  of  the  cost,  schedule  and  techni- 
cal "risks"  associated  with  computer  software.  Management  at 
all  levels  appears  to  be  taking  a hard  look  at  the  software 
developmental  process  in  an  effort  to  find  ways  to  better  con- 
trol that  process. 

2.  In  his  article  "Why  Projects  Fail",  Stephen  Keider  says  "One 
of  the  primary  causes  for  the  failure  of  data  processing  projects 
is  that  such  projects  are  often  not  initially  defined. . .Once  a 
project  has  begun,  no  one  seems  to  know: 
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. how  the  project  was  started; 

. what  the  staffing  is,  or  was,  at  any  point  in  time; 
. what  activities  have  been  performed; 


I 

I 

\ 
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. when  the  project  will  end; 

. what  the  project  will  accomplish. 

Essentially,  because  projects  are  rarely  formally  defined,  they 
are  rarely  completed."  (4:275). 

3.  There  is  still  another  factor  which  contributes  to  problems 
associated  with  computer  software  development.  That  problem  is 
the  same  one  that  raight  be  applied  to  almost  any  endeavor — 
management  visibility  and  control.  Perhaps  if  management  had 
control  of  and  visibility  into  the  development  process,  the 
problems  in  the  above  paragraphs  would  be  minimized.  Major 
Cecil  Martins  visited  many  Automatic  Data  Processing  installa- 
tions in  the  government,  private  industry  and  universities  com- 
piling data  for  a PHD  dissertation.  In  a presentation  summariz- 
ing his  findings,  he  stated  that  poor  management  visibility  and 
control  are  to  blame  for  most  of  the  problems  which  pervade  the 
system  development  process.  Further,  this  research  indicated 
that  those  organizations  which  have  adopted  an  orderly,  step- 
by-step  approach  to  software  development  exhibit  fewer  problems 
than  do  those  who  have  not  "disciplined"  the  effort  (13) . 

4.  The  thesis  of  this  paper  is  based  upon  the  aforementioned 
problems.  Namely,  that  management  visibility  and  control  are 
somewhat  laclcing  and  that  that  lac)c  allows,  even  encourages, 
other  problems.  It  is  further  argued  that  conventional  manage- 
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merit  practices  can  cope  with  the  software  development  process 
once  the  process  is  recognized  as  being  composed  of  several  com- 
ponent parts,  each  with  a purpose,  which  together  constitute  the 
total  computer  software  developmental  process.  Recognizing  the 
component  parts  allows  the  manager  and  the  technician  to  trans- 
cend the  whole,  thereby  gaining  an  insight  which  allows  under- 
standing, does  away  with  "artistry",  and  allows  both  management 
and  technical  discipline. 

5.  A brief  discussion  of  how  and  why  the  software  developmental 
process  arrived  at  its  present  state  is  warranted  at  this  point. 
First,  the  computer  industry  is  relatively  infantile  in  terms  of 
years.  However,  its  growth  has  been  phenomenal.  People  in  the 
trade  often  obtained  new  hardware  even  before  they  had  become 
accustomed  to  the  old.  This  gave  rise  to  a rather  common  opin- 
ion that  conventional  management  practices  cannot  be  applied 
to  software  development.  This  opinion  which  still  persists  has 
its  roots  in  the  very  early  days  of  computers.  In  the  fifties 
and  early  sixties,  computer  programmers  and  systems  analysts  were 
generally  left  to  accomplish  their  work  in  any  style  and  manner 
each  chose.  They  were  generally  exempt  from  management  control. 
In  short,  design  and  programming  were  generally  free  style.  Many 
Automatic  Data  Processing  personnel  relished  this  opportunity  to 
be  artistic  and  individualistic  made  possible  by  the  near  abdica- 
tion of  traditional  management  responsibilities.  In  a 1973 
Datamation  article,  Boehme  quotes  an  "Air  Force  Decisionmaker" 
as  that  decisionmaker  refers  to  ADP  people  as  follows; 
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You  software  cjuys  are  too  much  like  the 

weavers  in  the  story  about  the  Emperor  and  his 
new  clothes.  When  I go  out  to  check  on  a soft- 
ware development  the  answers  I get  sound  like, 

'We're  fantastically  busy  weaving  this  magic  cloth. 

Just  wait  awhile  and  it'll  look  terrific.'  But 
there's  nothing  I can  see  or  touch,  no  numbers  I 
can  relate  to,  no  way  to  pick  up  signals  that 
things  aren't  really  all  that  great.  And  there 
are  too  many  people  I know  who  have  come  out  at 
the  end  wearing  a bunch  of  expensive  rags  or 
nothing  at  all  (1:48). 

6.  Still  another  author,  Richard  L.  Nolan  writes  the  following: 

Top  management  has  never  been  totally  comfor- 
table with  the  (EDP)  Electronic  Data  Processing  mana- 
ger. And  it  is  fair  to  say  that  the  EDP  manager  has 
not  always  completely  understood  the  rationale  for 
top  management's  actions  in  regard  to  EDP."  Nolan 
continues,  "Not  so  subtle  is  the  continuing  high 
attrition  rate  among  those  who  hold  top  EDP  spots. 

I have  estimated  that  annual  EDP  management  turn- 
over in  1972  was  from  40%  to  50%.  In  a more  recent 
12  month  period,  25%  to  35%  of  the  EDP  managers  in 
more  than  450  companies  were  replaced,  according  to 
a survey  conducted  by  a major  vendor  of  computed  hard- 
ware and  software  in  1975.  These  rates  compared  to 
10%  to  15%  turnover  rates  among  other  senior  managers 
(8 : 402) . 

7.  Perhaps  the  most  notable  result  of  this  brief  history  of  dis- 
organization, the  one  which  causes  most  of  today's  developmental 
problems,  is  that  no  standard  developmental  methodology,  based 
upon  a series  of  stages  or  phases  of  developmental  effort,  is  in 
widespread  use.  The  Nolan  article  mentioned  above  "talks"  at 
great  lengths  about  management  but  never  really  describes,  or 
mentions  what  it  is  that's  being  maiiaged.  Without  a standard 
developmental  methodology,  there  is  little  hope  of  gaining  the 
much  needed  management  visibility  and  control  over  the  software 
developmental  process. 
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Section  II 


Present  Situation 

1.  The  Air  Force  has  recognized  the  risks  involved  in  computer 
software  development  and  has  taken  steps  toward  resolution  with 
its  AFR  300-15  Automated  Data  System  Project  Management.  That 
document  transcends  the  "whole"  and  does  identify  the  component 
parts  of  the  software  developmental  process,  trying  to  make  them 
visible  and  controllable.  Essentially,  AFR  300-15  employs  the 
principles  and  concepts  of  program  management  (phases,  baselines, 
formal  reviews)  as  practiced  by  the  Department  of  Defense  in  the 
acquisition  of  weapons  systems. 

2.  The  application  of  program  management  principles  and  concepts 
to  computer  software  development  is  a significant  step  forward 
but  the  tailoring  of  those  principles  may  not  have  gone  far 
enough.  In  short,  there  seem  to  be  places  in  AFR  300-15  where 
the  logical  developmental  process  was  shaped  to  fit  program 
management  rather  than  the  reverse. 

3.  The  purpose  of  this  paper  is  to  propose  changes  to  AFR  300-15 
to  further  refine/adapt  the  ideals  of  phasing,  baselining,  re- 
viewing and  auditing  to  the  logical  and  natural  computer  soft- 
ware developmental  process. 


Section  III 

Recommended  Changes  to  AFR  300-15 
1.  This  section  contains  specific  proposed  changes  to  AFR  BOO- 
TS. At  the  top  of  each  change  is  a page  number  which  corresponds 
to  the  page  in  AFR  300-15.  AFR  300-15  is  included  as  attach- 
ment 1 for  reference  purposes.  Rationale  for  each  change  pre- 
ceeds  it,  usually  on  the  same  page  for  shorter  changes.  In  the 
case  of  massive  changes,  the  entire  paragraph  was  rewritten. 


Page  1 


Rationale ; This  is  a change  to  the  title/purpose  pege  of  the 
regulation.  This  change  will  alter  the  intent  of  the  entire 
document.  Currently,  this  page  states  that  this  document  pro- 
vides "...a  methodology  for  the  management  of  data  system  (ADS) 
projects."  This  document  is  far  more  than  a "methodology  of 
management".  This  document  prescribes  a "software  development 
methodology".  Once  that  software  developmental  methodology  is 
established,  traditional  management  principles  and  practices  be- 
come workable. 

Change : Remove  the  first  sentence  and  insert  "This  regulation 

outlines  policies  and  procedures  which  provide  a standard  soft- 
ware developmental  methodology.  Management  of  that  methodology 
is  enhanced  through  a visible  developmental  life  cycle,  formal 
reviews,  baselines,  and  change  control  procedures.  It  must  be 
used  . . . (continue) " 
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Page  6 
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V. 


Rationale ; This  chapter  is  essentially  an  introduction  to  the 
document.  There  is  only  one  recommended  change  to  this  chapter. 
The  rationale  is  the  same  as  that  stated  for  the  change  to  page 

1. 

Change : Replace  paragraph  1-1  with:  "The  purpose  of  this  regu- 

lation is  to  prescribe  a standardized  software  developmental 
methodology.  The  software  developmental  life  cycle  is  fully 
described  in  terms  of  phases  and  associated  deliverable  products. 
Management  of  the  software  developmental  life  cycle  is  greatly 
enhanced  by  the  requirement  for  formal  reviews,  baseline  estab- 
lishment, and  other  factors  designed  after  the  engineering  dis- 
cipline of  configuration  management." 


■>- 


PROPOSED  CHAPTER  2 
OF  AFR  300-15 


Note:  Entire  Chapter  Rewritten 
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Chapter  2 
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Rationale;  This  is  a key  chapter  in  that  it  describes  the  soft- 
ware developmental  process  in  detail.  In  short,  this  chapter 
establishes  the  entire  software  developmental  cycle  by  identify- 
ing, for  the  first  time,  the  component  parts  or  anatomy  of  the 
developmental  life  cycle.  This  entire  chapter  was  rewritten 
and  with  it  the  philosophy  of  the  regulation  changes  from  a 
mere  management  document  to  a software  development/procedural 
document.  The  following  changes  were  made: 

a.  Phase  activities  realigned.  Several  phase  activities 
were  realigned  so  that  they  now  occur  in  an  entirely  different 
phase.  Other  activities  were  changed  or  replaced  which  forces 
more  discreteness  between  phases.  It  is  here  that  the  process 
takes  on  some  discipline  and  management  achieves  control  and 


visibility. 

b.  Reviews  were  renamed,  realigned  and  a new  one  was 
added.  Some  formal  reviews  had  names  which  were  short  on  mean- 
ingfulness in  the  data  processing  community.  The  names  were 
changed  to  more  accurately  reflect  their  function.  Secondly, 
some  were  realigned  to  occur  at  a different  point  in  the  cycle. 

c.  Baselines.  Baselines  were  moved,  renamed  and  a new 
one  was  added.  Like  the  reviews,  names  were  not  meaningful. 

They  were  renamed  to  more  clearly  reflect  their  purpose.  A 
baseline  was  added  after  the  concept  phase  as  none  was  there  be- 
fore. It  is  as  important  to  baseline  this  phase  as  it  is  to 
baseline  any  other. 
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CHAPTER  2 

SOFTWARE  DEVELOPMENT  METHODOLOGY 

2-1.  Purpose.  The  purpof3e  of  this  paragraph  is  to  describe  the 
software  development  methodology.  Essentially,  that  methodology 
consists  of  movement  through  a series  of  phases.  Each  phase  has 
a purpose  and  fulfillment  of  that  purpose  is  the  basis  for  action 
in  the  next  phase.  Baselines  serve  the  purpose  of  establishing 
the  fact  that  specific  actions  have  been  accomplished,  that 
change  control  procedures  are  to  be  used  thereafter,  and  to 
serve  as  the  point  of  departure  for  further  effort.  Baselines 
are  established  or  denied  at  reviews. 

2-2.  CONCEPTUAL  PHASE.  The  primary  purposes  of  the  actions  in 
this  phase  are  to  fully  document  the  requirement  at  conceptual 
level  and  to  gain  approval  to  proceed  with  further  exploration 
of  the  problem  or  definition  requirement.  This  approval  is 
concept  certification,  which  means  thcit  the  approval  authority 
(APR  300-2  approval  levels)  concurs  with  the  conceptualized 
solution  to  the  stated  problem  or  requirement. 

a.  TASKS.  The  first  task  is  the;  preparation  of  the  Data 
Automation  Requirement  as  per  the  instruction  in  AFR  300-12. 

In  order  to  prepare  this  document,  feasibility  studies,  func- 
tional systems  analysis,  etc.  may  be  undertaken.  Care  must  be 
taken  to  guard  against  designing  a software  system  at  this 
point.  Secondly  this  document  should  go  into  only  the  detail  re- 
quired to  gain  concept  certification.  Full  definition  comes  in  tlie 
next  phase.  The  approval  authority  will  issue  the  Data  Project 
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Directive.  The  purpose  of  this  document  is  to  formalize  concept 
certification  and  to  direct  that  the  project  proceed  through  sub- 
sequent phase (s). 

b . DOCUMENTS : 

(1)  Data  Automation  Requirement 

(2)  Data  Project  Directive 

c.  REVIEW;  The  Concept  Certification  Review  will  be  per- 
formed at  the  conclusion  of  this  phase.  It  may  be  performed  by 
the  APR  300-2  approval  authority  or  at  a level  prescribed  by 

him.  The  results  of  the  review  and  resultant  recommendations  will 
be  forwarded  to  the  APR  300-2  approval  authority  for  final 
approval  and  granting  of  concept  certification.  In  this  case, 
a letter  of  concept  certification  will  be  signed  by  the  approval 
authority  and  the  letter  will  be  appended  to  the  DAR.  Any  DAR 
without  such  a letter  will  be  considered  as  not  having  been  cer- 
tified. Concept  certification  automatically  establishes  the 
Concept  Baseline  and  change  control  procedures  automatically  are 
invoked  relative  to  the  DAR. 

2-3.  DEPINITION  PHASE.  The  purpose  of  the  Definition  Phase  is 
to  fully  define  the  functional  aspects  of  the  system  to  be  de- 
veloped. This  is  not  to  be  construed  as  a software  system  design, 
which  is  the  purpose  of  the  next  phase.  It  is  the  purpose  of 
this  phase  to  take  the  DAR  prepared  in  the  previous  phase  as  the 
basis  and  prepare  a Functional  Description  which  requires  the 
most  detailed  functional  definition  of  the  requirement.  It  is 
logical  that  the  software  system  be  conceptualized  during  this 
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phase  and  the  Functional  Description  provides  for  that. 

a.  TASKS 

(1)  Prepare  the  Data  Project  Plan 

(2)  Prepare  the  Functional  Description 

(3)  Begin  drafting  user,  operation  and  maintenance 

manuals 

b.  Documents;  (as  specified  in  a.  cibove) 

c.  Review.  The  Functional  System  Definition  Review  is  per- 
formed at  the  close  of  this  phase.  The  purpose  of  this  review 

is  to  assure  that  the  Functional  Description  portrays  the  func- 
tional specifications  to  the  satisfaction  of  the  user.  Secondly, 
the  ADP  systems  analysts  must  agree  that  the  information  pro- 
vided is  sufficient  to  serve  as  the  basis  for  system  design. 
Approval  of  the  Functional  Description  automatically  establishes 
the  Functional  Baseline  and  invokes  change  control  procedures 
against  the  Functional  Description. 

2-4.  DEVELOPMENT  PHASE.  The  purpose  of  this  phase  is  to  design 
the  software  system  and  to  design,  code  and  test  the  computer 
programs.  Accordingly,  this  phase  is  divided  into  two  segments: 
design  and  program.  The  Design  Baseline  separates  the  two 
segments . 

a.  Design  Segment  Tasks  and/or  Documents; 

(1)  Prepare  the  system/subsystem  specifications 
(constitutes  the  design  of  the  software  system) 

(2)  Prepare  Data  Requirements  Document 

(3)  Prepare  Data  Base  Specifications 
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b.  Review:  The  System  Design  Review  is  performed  to  deter- 
mine the  adequacy  of  the  system  design.  The  capabilities  of  the 
design  must  accomplish  the  requirements  established  in  the  Func- 
tional Description.  Upon  favorable  completion  of  this  review, 
the  Design  Baseline  is  automatically  established.  The  system/ 
subsystem  specifications  are  then  covered  by  change  control 
procedures . 

c.  Program  Segment  Tasks  and/or  Documents 

(1)  Prepare  Program  specifications 

(2)  Perform  Program  Logic  Review  iteratively  until 
each  program  is  approved  for  coding 

d.  Reviews. 

(1)  A Program  Logic  Review  is  performed  on  each  Pro- 
gram Specification  prior  to  the  commencement  of  coding.  The  pur- 
pose is  tc  assure  the  adequacy  of  the  program  logic  and  to 
assure  that  the  program  will  perform  as  specified  by  the  system/ 
subsystem  specification. 

(2)  The  Product  Verification  Review  is  the  formal  cul- 
mination of  the  physical  and  functional  configuration  audits  per- 
formed throughout  this  phase.  Completion  of  this  review  estab- 
lishes the  Product  Baseline. 

2-5.  TEST  PHASE.  The  installation  and  validation  test  of  the 
system  in  the  operational  environment  by  the  designated  user 
representative  or  by  an  independent  agency. 

a.  VALIDATION  TESTING.  Conduct  two-stage  validation  test- 


ing of  the  system  by  an  independent  test  group.  In  the  first 


1 


stage,  perform  in-house  validation  testing  of  the  system  re- 
quirements, correct  problems,  and  update  documentation.  In  the 
second  stage,  rerun  validation  test  for  the  user,  or  review 
successful  in-house  validation  test  runs  with  the  user.  In  addi- 
tion, validate  program  specifications  and  other  relevant  documen- 
tation in  both  stages  (paragraph  6-5) . 

b.  SYSTEM  VALIDATION  REVIEW  (SVR) . The  SVR  validates  that 
performance  of  a CPCI,  as  determined  through  validation  testing, 
complies  with  the  SS  and  the  FD  (paragraph  4-9). 

2-6.  OPERATION  PHASE.  The  operation,  maintenance,  and  product 
improvement  by  the  user  and/or  the  developer. 
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Rationale.  Configuration  management  is  a relatively  new  term  to 
many  people  in  the  ADP  community.  Therefore,  a definition  of 
configuration  management  is  in  order  in  the  lead  paragraph  on 
that  subject.  This  change  does  that. 

Change . Remove  paragraph  3-1  and  replace  it  with  the  following 

paragraph.  "Configuration  management  is  a term  long  associated  ■ 

with  the  development  of  weapons  systems.  The  term  now  applies 

to  the  development  of  general  purpose  computer  software  as  the 

concepts  and  principles  of  configuration  management  have  been  ] 

i 

adapted  to  that  usage.  This  document  seeks  to  ensure  that  the  i 

software  developmental  life  cycle  is  orderly,  visible  to  manage-  ' 

i 

ment  and  users,  is  controllable  and  the  outcome  more  predictable.  | 

To  those  ends,  the  software  configuration  management  concepts 
of  phases,  baselines,  formal  reviews,  and  change  control  have 
been  adapted  and  included  herein.  While  the  concepts  and  the 
principles  are  the  same,  the  phases,  baselines,  reviews,  and 
change  control  concepts  have  been  revised,  shifted,  tailored  and 
otherwise  changed  to  make  them  "fit"  the  logical  software  de- 
velopmental life  cycle.  Within  the  AFR  300-series  software 
developmental  area  configuration  management  is  the  discipline 
applied  to  the  developmental  life  cycle  which  gives  it  a visible, 
comprehensible  structure  (phase),  enhances  control  (baselines, 
reviews)  and  which  enhances  developmental  changes  (change  con- 
trol) . 
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Chapter  4 


Rationale : The  changes  in  Chapter  2 incorporated  some  new  re- 

views. This  chapter  includes  the  details  for  accomplishing  re- 
views. There  is  no  additional  rationale  beyond  what  was  provided 
for  Chapter  2.  Review  additions  or  changes  are: 

4-3.  Concept  Certification  Review  - New 
4-4.  Functional  System  Definition  Review  - Name  change 
and  partial  change  in  function. 

4-6.  Program  Logic  Review  - Name  change 


CHAPTER  4 


Reviews  and  Audits 

4-1.  PURPOSE.  The  purpose  of  this  chapter  is  to  describe  the 
reviews  and  audits  that  must  be  performed  throughout  the  soft- 
ware developmental  life  cycle. 

4-2.  PARTICIPATION  AND  RESPONSIBILITIES.  The  project  manager 
or  his  designee  chairs  all  reviews  and  audits.  The  user  repre- 
sentative will  make  sure  that  the:  functional  requirements  are 
being  adequately  met. 

a.  Project  Manager  Responsibilities. 

(1)  Establish  the  time,  the  place,  and  the  agenda  for 
the  reviews  and  audits,  and  coordinate  these  arragements  with 
all  participants.  Depending  on  the  type  of  review  or  audit, 
schedules  for  incremental  reviews  may  have  to  be  prepared  and 
coordinated  with  all  participants. 

(2)  Prepare  for  each  review  or  audit  in  sufficient 
depth  consistent  with  project  scope  and  size.  Material  to  be 
reviewed  must  be  submitted  to  participants  prior  to  each  review, 
allowing  time  for  comment.  Documents  should  be  reviewed  for 
technical  rather  than  editorial  content.  Review  participants 
should  prepare  design  problem  reports  (DPRs)  and  submit  them  to 
the  originator  of  the  material  being  reviewed.  The  originators 
of  the  material  will  evaluate  the  DPRs  and  reply  to  them  at  the 
formal  review.  Another  approach  is  to  correct  minor  discrep- 
ancies prior  to  the  review,  advise  participants  of  the  disposi- 
tion, and  hold  the  major  DPRs  for  discussion  at  the  meeting. 
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b.  Resolution  of  disagreements  betvaen  the  development 
activity  and  the  user  will  be  escalated  to  the  next  level  of  man- 
agement for  resolution. 

4-3.  CONCEPT  CERTIFICATION  REVIEW  (CCR) . The  CCR  is  conducted 
during  the  conceptual  phase  to  formally  approve  the  concept  by 
review  and  approval  of  the  DAR. 

a.  Review  Items.  The  principal  items  to  be  reviewed  at 
the  CCR  are  the  Data  Automation  Requirement  (DAR) , and  the  Data 
Project  Directive  (DPD) . 

b.  Postreview  Action.  After  completion  of  the  CCR,  the 
minutes  are  distributed  by  the  project  manager.  Notification 
of  any  required  postreview’  action  items,  plus  official  acknow- 
ledgment of  conduct  and  completion  of  the  review,  are  provided 
to  the  user,  the  developer,  and  the  appropriate  ADP  approval 
authority. 

4-4.  FUNCTIONAL  SYSTEM  DEFINITION  REVIEW  (FSDR) . The  FSDR 
establishes  the  functional  baseline  for  individual  CPCIs  or  for 
the  entire  system. 

a.  Review  Items. 

(1)  The  FSDR  documentation  package  should  include  the 
following : 

(a)  Functional  Description  (FD)  with  changes 

(b)  Data  Project  Plan  (DPP) 

(2)  The  FSDR  is  conducted  to  accomplish: 

(a)  Review  of  the  FD 

(b)  Insure  the  technical  project  risks  are  iden- 
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tified,  ranked,  avoided,  and  adequate  tradeoffs  are  made. 

(c)  Insure  a technical  understanding  of  require- 
ments has  been  reached  and  technical  direction  is  provided  to 
project  participants. 

(d)  Insure  project  schedules  and  milestones  are 
consistent  with  the  allocated  requirements. 

b.  Postreview  Action.  After  completion  of  the  FSDR,  the 
minutes  of  the  meeting  are  coordinated  with  all  participants. 
Participants  will  also  be  advised  of  any  postreview  action 
items  requiring  response.  Approval  of  the  FD  establishes  the 
functional  baseline. 

4-5.  SYSTEM  DESIGN  REVIEW  (SDR).  The  SDR  is  a technical  review 
of  the  system/subsystem  design  for  a computer  program  configura- 
tion item  (CPCI) . Only  one  successful  SDR  per  CPCI  is  required. 
A collective  SDR  for  a functionally  related  group  of  CPCIs, 
treating  each  CPCI  individually,  may  be  held  when  advantageous 
to  the  project.  The  overall  technical  risks  associated  with 
each  CPCI  are  also  reviewed  on  a technical,  cost,  and  schedule 
basis.  NOTE:  If  the  ADS  is  defined  as  a single  CPCI,  the  SDR 
may  be  combined  with  the  FSDR. 
a.  Review  Items. 

(1)  The  SDR  documentation  package  should  include  the 
following : 

(a)  Updated  functional  description  (FD) 

(b)  System/subsystem  specification  (SS) 

(c)  Draft  of  CPCI  test  plans  (PT) , less  test 


21 


procedures 


(d)  Data  base  structure  and  organization 

(e)  Design  problem  reports  (DPR) 


The  SDR  is  conducted  to  establish  the  following: 

(a)  Compatibility  of  the  design  with  the  SS  for 
with  other  CPCIs  with  which  it  must  interface 

(b)  Adequacy  of  test  plan 

To  satisfy  the  intent  of  the  SDR,  the  following 
be  performed 

(a)  Review  all  functional  interfaces.  Review 
message  formats,  storage  availability,  and  other  considerations 
that  may  have  been  established  in  the  SS.  At  this  time,  the 
interfaces  between  CPCI  and  hardware  CIs  should  be  defined  at 

a level  low  enough  to  satisfy  all  software  development  needs. 

(b)  Review  the  structure  of  the  CPCI  as  a whole, 
with  emphasis  on  allocation  of  the  functions  delineated  in  the 
SS  to  individual  computer  programs;  storage  requirements  and 
allocation;  computer  program  operating  sequences;  and  design  of 
the  data  base 

(c)  Analyze  critical  timing  requirements  to  insure 
the  proposed  CPCI  design  satisfies  the  timing  requirements 

(d)  Review  human  factors  impact  of  proposed  sub- 
sustem  design  of  the  CPCI 

(e)  Review  all  changes  to  the  SS  subsequent  to  the 


established  allocated  baseline.  Insure  the  FD  and  SS  adequately 
reflect  these  changes 


^ 


S 


(2) 

the  CPCI  and 

(3) 

tasks  should 


(f)  Review  the  PT  to  insure  it  satisfies  the  re- 
quirements of  the  FD  and  SS 

(g)  Review  status  of  all  negative  and  provisional 
entries  such  as  "not  applicable"  or  "to  be  determined"  in  the  SS 

b.  Postreview  Action.  After  completion  of  the  SDR,  the 
minutes  are  published  and  distributed  by  the  development  activity. 
The  user  will  be  notified  of  any  required  postreview  action. 

4-6.  PROGRAM  LOGIC  REVIEW  (PLR) . 

a.  Purpose.  The  PLR  is  a review  conducted  during  the  de- 
velopment phase  before  translating  logic  to  coded  instruc- 

tions. The  PLR  insures  the  detailed  design  solution,  as  reflected 
in  the  draft  program  specification  (PS) , satisfies  performance 
requirements  established  by  the  SS.  The  PLR  also  verifies 
system  design  compatibility  with  other  CPCIs,  and  within  the 
CPCI.  Approval  of  the  project  manager  is  required  to  release 
portions  of  CPCIs  for  coding  prior  to  PLR  when  necessary  to 
maintain  schedules. 

b.  Review  Items  and  Tasks 

(1)  The  PLR  documentation  should  consist  of  the 
following : 

(a)  System/subsystem  specification  (SS) , including 
approved  changes 

(b)  Program  specifications 

(c)  CPCI  test  plans  (PT) , including  test  proce- 
dures 

(d)  Design  problem  reports  (DPR) 


(e)  System  design  review  (SDR)  minutes 

(f)  Draft  of  computer  operation  manual  (OM) 

(g)  Draft  of  program  maintenance  manual  (MM) 

(h)  Draft  of  the  program  specification  (PS) 

4-7.  CONFIGURATION  AUDITS 

a.  Purpose.  A configuration  audit  is  a process  employed 
to  verify  conformance  of  a CPCI  to  specifications  and  standards. 
Two  audits,  the  Functional  Configuration  Audit  (FCA)  and  the 
Physical  Configuration  Audit  (PCA) , are  required  at  two  points 
during  the  ADS  life  cycle  for  an  ADS  requiring  greater  than 
fifty  development  manyears.  The  first  point,  prior  to  the 
Product  Verification  Review  (PVR) , is  termed  the  preliminary  FCA 
and  PCA,  and  the  second  point  is  prior  to  the  System  Validation 
Review  (SVR)  and  termed  tht;  final  FCA  and  PCA.  The  preliminary 
audits  are  a thorough  technical  examination  of  the  CPCI,  while 
the  final  audits  are  directed  at  changes  made  to  the  CPCI  as  a 
result  of  formal  testing.  The  following  guidelines  for  conduct- 
ing the  FCA  and  PCA  are  provided. 

b.  FUNCTIONAL  CONFIGURATION  AUDIT  (FCA) . The  FCA  validates 
the  CPCIs  actual  performance  compliance  with  the  SS.  Test  data 
is  reviewed  to  validate  that  the  CPCI  performs  as  required.  The 
project  manager  provides  two  FCA  data  packages  to  the  organiza- 
tion performing  the  FCA,  one  to  be  delivered  at  some  negotiated 
lead  time  prior  to  FCA,  and  the  other  to  be  made  available  during 
the  FCA. 
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(1)  Items  to  be  provided  before  audit  are; 

(a)  Nomenclature  (name  or  descriptive  title  of 

the  CPCI) 

(b)  Identification  of  necessary  documentation 

(2)  The  documentation  package  to  be  made  available 
for  FCA  consists  of: 

(a)  Program  specification  (PS) , including  list- 
ings 

(b)  System/subsystem  specification  (SS)  with 
changes,  if  applicable 

(c)  Test  plan  (PT)  and  when  available,  test  re- 
ports (RT) 

(d)  PLR  and  CDR  minutes 

(e)  Status  accounting  records  of  SPRs,  DBCRs, 

and  BCRs 

(3)  The  FCA  consists  of  examining  and  evaluating: 

(a)  Test  procedures  and  results  to  determine  CPCI 
conformance  with  the  SS  and  FD  and  quality  assurance  provisions 

(b)  Adequacy  of  analysis  or  simulations  where 
performance  parameters  cannot  completely  be  verified  by  test 

(c)  Any  requirements  stated  in  the  FD  that  could 
not  be  met,  and  the  developer's  proposed  solution 

(d)  All  approved  BCRs  to  insure  they  are  incor- 
porated and  verified  during  testing 

(4)  Deficiencies  noted  and  recommended  corrective  ac- 
tions are  documented  in  the  FCA  minutes. 

25 


A 


c. 


PHYSICAL  CONFIGURATION  AUDIT  (PCA) 


The  PCA  is  a for- 


mal examination  of  the  coded  version  of  a CPCI  against  its  tech- 
nical documentation.  The  PCA  includes  a selective  audit  of  the 
program  specification  (PS),  including  logic  charts,  listings,  and 
design  narrative.  It  also  includes  a review  of  the  formal  and 
completeness  of  any  other  documentation  due  for  acceptance.  The 
project  manager  provides  two  PCA  data  packages  to  the  organiza- 
tion performing  the  PCA  - one  to  be  delivered  at  some  negotiated 
lead  time  prior  to  PCA,  and  the  other  to  be  made  available  dur- 
ing PCA. 

(1)  Items  to  be  delivered  prior  to  audit  consist  of 

(a)  PCA  date  and  location 

(b)  Identification  of  CPCIs  to  be  accepted 

(2)  The  documentation  package  to  be  provided  and  made 
available  for  PCA  consists  of: 

(a)  System/subsystem  specification  (SS) 

(b)  Program  specification  (PS) 

(c)  Test  procedures  (PT)  and  test  report  (RT) 

(d)  Draft  computer  operation  manual  (OM) 

(e)  Draft  program  maintenance  manual  (MM) 

(f)  CPCI  tape  and  listings 

(g)  Applicable  configuration  management  records 
c.  The  PCA  consists  of  reviewing: 

(1)  PS  for  format  and  completeness 

(2)  Selectively  compare  the  listings  with  documentation 

(3)  Computer  operation  manual  (OM)  and  program  mainten- 


ance  manual  (MM)  for  format  and  completeness 

(4)  The  release  and  change  control  system  to  be  sure 
it  can  properly  control  the  processing  and  formal  release  of 
changes . 

4-8.  PRODUCT  VERIFICATION  REVIEW  (PVR). 

a.  Purpose.  PVR  is  conducted  for  each  CPCI  at  the  end  of 
the  development  phase  to  establish  the  Product  Baseline  for  that 
CPCI  and  to  insure  preparation  for  the  test  phase  has  been  com- 
pleted. 

b.  A preliminary  functional  configuration  audit  (FCA)  and 
a preliminary  physical  configuration  audit  (PCA)  will  be  con- 
ducted prior  to  the  PVR  to  provide  management  with  an  independent 
technical  assessment  of  the  CPCI. 

c.  Items  reviewed  at  the  PVR  include; 

(1)  Findings  and  recommendations  resulting  from  the 
preliminary  FCA  and  PCA 

(2)  Test  plans 

(3)  Preparations  and  schedules  for  the  test  phase 

d.  Discrepancies  which  would  lead  to  establishing  an  in- 
valid product  baseline  and  premature  testing  must  be  corrected 
prior  to  successful  completion  of  the  PVR,  Completion  of  the  PVR 
establishes  the  CPCI  product  baseline  and  initiates  the  formal 
test  phase  as  described  in  Chapter  6. 

4-9.  SYSTEM  VALIDATION  REVIEt'f  (SVR)  . The  purpose  of  the  SVR  is 
to  review  the  results  of  validation  testing  to  insure  that  the 
ADS  satisfies  the  requirements  of  the  SS  and  the  FD.  The  SVR  is 
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supported  by  the  final  FCA  and  PCA. 

a.  Review  items.  The  following  items  are  reviewed  at  the 

SVR. 

(1)  PVR  minutes 

(2)  Findings  and  recommendations  resulting  from  the 
final  FCA  and  PCA 

(3)  Test  Report  (RT) 

(4)  Technical  documentation  as  required 

(5)  Configuration  management  records  as  required 

b.  A prerequisite  for  successful  completion  of  the  SVR  is 
the  certification  by  the  functional  OPR  or  designated  functional 
representative  that  the  ADS  adequately  satisfies  the  requirements 
stated  in  the  FD.  Completion  of  the  SVR  terminates  the  test 
phase  and  initiates  the  operation  phase. 


28 


I 


AFR  300-15 


AUTOMATED  DATA  SYSTEM  PROJECT  MANAGEMENT 


June  1977  (Draft) 


Attachment  1 


1 


DEPARTMENT  OF  THE  AIR  FORCE 


AF  REG  :^TION  300-1 b 


! 


r 

I 


I 

I 


Headquarters  US  Air  Force 
Washington,  DC  20330 

Data  Automation 

AUTOMATED  DATA  SYSTEM  PROJECT  MANAGEMENT 
This  regulation  outlines  policies  and  procedures  which 
provide  a methodology  for  the  management  of  data  system 
(ADS)  projects.  It  must  be  used  with  AFRs  300-2,  300-7,  and 
300-12.  It  applies  to  all  Air  Force  activities  responsible 
for  planning,  designing,  developing,  authorizing,  selecting, 
acquiring,  maintaining,  and  managing  ADS  projects.  It 
implements  DOD  Manual  4120. 17M,  December  29,  1972  and  AFR 
65-3,  July  1,  1974. 


OPR:  AF/KRAX  (Maj  M.D.  Hall) 

KRA  APPROVAL:  Brig  Gen  Frederick  L.  Maloy 
WRITER  EDITOR: 

DISTRIBUTION:  F 


± 


TABLE  OF  CONTENTS 


Paragraph 


Applicability  

CHAPTER  2 - OVERVIEW  OF  ADS  PROJECT  MANAGEMENT 

Purpose  

Conceptual  Phase  

Definition  Phase  

Development  Phase  

Test  Phase  

Operation  Phase  

Project  Management  Functions  

CHAPTER  3 - CONFIGURATION  MANAGEMENT 

Purpose  

Configuration  Management  Plan  

Configuration  Identification  

Configuration  Control  

Configuration  Status  Accounting  - 


1-1 

1-2 

1-3 

1-4 


Page 


L 

(o 


2-1 

2-2 

2-3 

IZ- 

2-4 

1 3 

2-5 

) 7 

2-6 

;7 

2-7 

n 

3-1 

Zo 

3-2 

7.0 

3-3 

P.O 

3-4 

3-5 

3^ 

2. 


CHAPTER  4 - REVIEWS  AND  AUDITS 


Paragraph 


Page 


1 

i- 


Purpose  

Participation  and  Responsibilities 
System/Subsystem  Requirements 

Review  (SRR)  

System  Design  Review  (SDR)  

Preliminary  Design  Review  (PDR)  — 

Critical  Design  Review  (CDR)  

Configuration  Audits  

Product  Verification  Review 

(PVR)  

System  Validation  Review  (SVR)  — 
CHAPTER  5 - QUALITY  ASSURANCE 

Purpose  

Definition  

Function  

Planning  

Policies,  Procedures  and 

Standards  

Configuration  Audits  

QA  Participation  in  Reviews  


4-1  31 

4-2  3 7 


4-3 

37 

4-4 

37 

4-5 

41 

4-6 

43 

4-7 

44 

4-8 

47 

4-9 

4? 

5-1 

S-O 

5-2 

s^o 

5-3 

S'o 

5-4 

5-5 

S'^ 

5-6 

£-/ 

5-7 

Si 

A 


Test  Phase  Support 


•t 

T. 


CHAPTER  6 


TEST  MANAGEMENT 


Paragraph 


5-8 


j 


Page 


1 

■1 


General  

Purpose  

Information  

Development  Testing  

Validation  Testing  

Environmental  System  Test,  Phase 

I (EST-I)  

Environmental  System  Test,  Phase 

II  (EST-II)  

ATTACHMENTS 

1 - Glossary  

2- Development  Test  Plan  

3- Program  Maintenance  Manual  

4- Configuration  Management  Plan  (CMP) 

5- Configuration  Management  Logs  

6- Change  Control  Forms  


6-1 

6~3 

6-2 

i"3 

6-3 

5-3 

6-4 

s-f 

6-5 

SS’ 

6-6 

SI 

6-7 

lo\ 

71 

73 

75" 

79 

i?f 


J 

\ 


FIGURES: 


: ^ 
: s 


Page 


1- 1  Documentation  and  Review  Determination  % 

2- 1  Automated  Data  System  Life  Cycle /£) 

3- 1  Configuration  Change  Control  Phasing  During 

Project  Lifetime 2.7 


Chapter  1 

.* . INTRODUCTION 

W 

V. 

1-1.  PURPOSE:  To  prescribe  a methodology  for  the  management 
of  automated  data  system  (ADS)  projects.  This  regulation 
provides  guidelines  for  organizing,  planning  developing  and 
maintaining  an  ADS  using  the  engineering  disciplines  of 
baseline  configuration  management,  reviews  and  audits,  and 
quality  assurance. 

1-2.  TERMS:  Terms  used  in  this  regulation  are  defined  in  APR 
300-2,  AFR  300-12,  Volume  I,  and  Attachment  1,  this  regulation. 
The  heirarchy  of  terms  used  in  this  regulation  to  describe 
computer  programs  are  Automated  Data  System  (ADS),  Computer 
Program  Configuration  Item  (CPCI) , and  module.  The  relation- 
ship of  ADS  and  CPCI  is  described  in  paragraph  3-3a. 

1-3.  OBJECTIVE:  To  provide  guidance  for  the  management  of  ADS 
projects . 

1-4.  APPLICABILITY:  The  principles  and  concepts  of  con- 
figuration management  apply  to  ADSs  developed  under  the  300- 
series  Air  Force  Regulations.  The  following  guidance  is 
provided  to  aid  in  the  use  of  this  regulation. 

a.  The  disciplines  described  are  applicable  to  all  ADS 
projects.  The  procedures  will  be  applied  in  their  entirety 
to  those  systems  which  require  more  than  50  man-years 
development  effort.  For  systems  requiring  50  man-years  or 
less,  the  degree  to  which  the  procedures  apply  will  be 
consistent  witji  _the  system's  cost  and  complexity. 

(o 


r: 


c.  A single,  common  set  of  configuration  management 

T procedures  and  quality  assurance  procedures  may  not  meet  the 

needs  of  all  Air  Force  organizations.  These  procedures 
V should  be  tailored  to  recognize  variations  in  requirements, 

organizations,  and  working  relationships.  MAJCOM/SOA 
5?-  supplements  to  this  regulation  will  be  prepared  if  required. 

t 

d.  Any  organizational  alignments  which  may  be  implied 
in  this  regulation  are  not  mandatory. 

e.  The  project  manager  will  establish  the  degree  of 
formality  of  reviews  and  audits  consistent  with  the  size  of 
the  project. 
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2-1.  PURPOSE.  To  introduce  and  briefly  define  the  docu- 
ments that  are  prepared,  the  project  management  functions 
that  eire  performed,  the  development  discipline  that  is 
followed,  and  the  reviews  and  audits  necessary  to  insure  the 
successful  application  of  baseline  conf igiiration  management 
techniques  to  ADS  projects.  (See  Figure  2-1.) 

2-2.  CONCEPTUAL  PHASE.  A determination  of  mission  and  sys- 
tem requirements.  This  regulation  describes  those  ADS 
development  events  which  occur  after  the  conceptual  phase 
and  formal  approval  of  the  Data  Automation  Requirement 
(DAR) . Procedures  for  initial  analysis,  program  adversary/ 
advocacy  actions,  requirements  approval  and  management 
planning  are  provided  by  AFRs  300-2,  300-12,  Volume  I,  and 
300-7. 

a.  TASKS. 

(1)  IDENTIFICATION  OF  A REQUIREMENT.  A functional 
manager's  description  and  justification  of  a "need"  for  a 
capability  to  meet  a mission  or  operational  requirement  is 
the  beginning  of  the  conceptual  phase.  The  user  must  inject 
his  expertise  into  development  of  a descriptive  definition 
of  the  initial  system  concept. 

(2)  REQUIREMENTS  ANALYSIS.  Requirements  analysis  is 

the  identification  and  definition  of  the  problems  associated  | 

i 

with  providing,  changing,  or  converting  a management  or  | 

! 


( 


operational  capability  that  will  best  meet  the  mission  of  an 
organization.  The  set  of  requirements  that  emerges  from  a 
requirements  analysis  is  the  main  device  used  to  achieve 
project  direction  and  control.  The  inability  to  produce 
this  governing  set  of  requirements  may  be  e.n  indicator  of  a 
failing  project.  Thus,  it  is  essential  that: 

(a)  Requirements  analysis  be  thorough  and  avoid 
specifying  a design  solution. 

(b)  Requirements  analysis  documentation  contain 
a complete  and  understandable  definition  of  the  require- 
ments . 


(c)  Compliance  with  regulatory  requirements 
dealing  with  policies  and  procedures  be  enforced. 

(3)  PREPARATION  OF  DOCUMENTATION.  Once  the  user  has 
identified  a requirement,  evaluated  alternative  solutions, 
documented  requirements  for  management  evaluation,  and 
automation  is  selected  as  the  solution,  certain  documents 
must  be  prepared. 

b.  DOCUMENTS.  Documents  prepared  during  the  conceptual 
phase  include  the  Data  Automation  Requirement  (DAR) , the 
Data  Project  Directive  (DPD) , the  Data  Project  Plan  (DPP) , 
and  the  Functional  Description  (FD) . The  DAR,  DPD,  DPP,  and 
FD  preparation  requirements  are  specified  in  AFR  300-12, 
Volume  I. 

C.  SYSTEM/SUBSYSTEM  REQUIREMENTS  REVIEW  (SRR) . The  SRR 
is  conducted  during  the  conceptual  phase  to  assure  the  user 
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that  the  project's  progress  is  responsive  to  the  approved 
requirement  and  to  determine  initial  direction  of  the  design 
effort  (paragraph  4-3) . 

2-3.  DEFINITION  PHASE.  In  this  phase  one  or  more  system 
developers  define  the  design  requirements  for  the  major  ele- 
ments in  the  system. 

a.  TASKS. 

(1)  DEVELOP  SYSTEM  INTERFACE  CONTROL  REQUIREMENTS. 
Define  interface  between  operational  functions.  Include: 

(a)  Initial  performance  allocation. 

(b)  Sequencing/scheduling. 

(c)  Control  techniques. 

(2)  EXPAND  SYSTEM  REQUIREMENTS.  Perform  a compre- 
hensive and  critical  review  of  functions,  performance,  and 
design  requirements,  and  expand  them. 

(3)  DEFINE  COMPUTER  PROGRAM  REQUIREMENTS.  Define 
requirements  to  be  satisfied  by  the  operational  computer 
programs.  This  definition  should  include  both  narrative  and 
graphic  presentation  of  information  flow. 

(4)  DEFINE  EQUIPMENT  PERFORMANCE  REQUIREMENTS. 

Define  requirements  for  the  tasks  to  be  performed  by  the 
computer  and  other  system  hardware. 

(5)  PREPARE  DOCUMENTS. 

b.  DOCUMENTS. 

(1)  SYSTEM/SUBSYSTEM  SPECIFICATIONS  (SS) . The  SS  is 
a technical  document  that  governs  the  development  of  an  ADS 


or  subsystem  of  an  ADS.  System/Subsystem  Specifications 
define  performance,  interfaces,  and  other  technical  require- 
ments sufficiently  to  permit  detailed  design.  Approval  of 
this  docioment  at  the  System  Design  Review  (SDR)  establishes 
the  allocated  baseline  for  the  system  or  subsystem. 

(2)  DATA  REQUIREMENTS  DOCUMENT  (RD) . A document 
that  specifies  the  characteristics  and  limitations  of 
required  data.  Defines  inputs  required  of  the  user,  proce- 
dures for  providing  this  input  to  the  system  files,  expected 
outputs,  use  of  standard  data  elements  and  codes,  and  data 
limitations. 

(3)  DATA  BASE  SPECIFICATION  (DS)  . This  docxament 
specifies  the  design  of  the  data  base  and  establishes  the 
definitions  of  the  interfaces. 

c.  SYSTEM  DESIGN  REVIEW.  The  SDR  is  a review  of  the 
total  system  requirements  used  to  produce  the  system  defini- 
tion (system/subsystem  specifications) . Successful  com- 
pletion terminates  the  definition  phase  (paragraph  4-4) . 

2-4.  DEVELOPMENT  PHASE.  Analysis  and  design,  coding, 
debugging,  integration,  and  development  testing  of  computer 
program  configuration  items  (CPCI)  is  done  in  this  phase. 

a.  TASKS.  The  sequence  of  tasks  during  the  development 
phase  is: 

(1)  REVIEW  SYSTEM/SUBSYSTEM  SPECIFICATIONS  (SS) . 
Perform  a review  of  the  subsystem  design.  When  the  ADS  is 
defined  as  a single  CPCI,  the  System  Design  Review  (SDR)  and 
Preliminary  Design  Review  (PDR)  can  be  combined. 


(2)  PRELIMINARY  PROGRAM  DESIGN. 

(a)  Define  functions  which  will  be  performed  by 
individual  programs. 

(b)  Complete  definition  of  inputs,  processing, 
and  outputs  of  programming  tasks.  When  defined,  assign  to 
specific  modules  and  develop  refined  functional  flows.  A 

I 

* ! 

system  simulation  study  may  be  necessary  to  avoid  the  risk 

of  serious  design  deficiencies.  i \ 

(3)  INITIATE  SUPPORT  DOCUMENTATION.  Begin  prelimi- 

i 

( 

nary  drafts  of  the  Program  Maintenance  Manual  (MM)  , Users 
Manual  (UM) , Computer  Operation  Manual  (OM) , and  Test  Plan 
(PT)  . 

(4)  DETAIL  FUNCTION  DESIGN.  Expand  flowcharts, 
hierarchical  input/ process/output  (HIPO)  charts,  structure 
charts,  or  other  logic  designs,  algorithms,  and  narrative 
descriptions  in  sufficient  detail  to  provide  the  basis  for 
actual  coding.  Complete  definition  of  the  data  base  at  the 
module  level,  including  number,  type,  and  structure  of 
tables  and  description  of  items  in  the  tables.  Define  all 
elements  as  dynamic  or  static,  and  their  uses. 

(5)  CONDUCT  CRITICAL  DESIGN  REVIEW.  Conduct  a user 
and  developer  review  to  confirm  that  the  design  meets  its 
functional  development  requirements,  and  that  the  design  is 
sufficiently  defined  to  permit  the  start  of  coding.  Approval 
of  program  specifications  (PS)  at  the  Critical  Design  Review 
(CDR)  finalizes  the  design  of  the  ADS. 


(6)  CODE,  COMPILE,  AND  VERIFY.  Code,  conpile  and 
verify  modules.  Conduct  development  testing  of  coded 
modules  until  complete  programs  and  systems  are  developed 
and  verified.  Development  programmers  are  responsible  for 
development  testing. 

b.  DOCUMENTS.  Documents  initiated  in  previous  phases 
are  updated,  and  the  following  documents  are  initiated. 

(1)  TEST  PLAN  (PT) . A document  that  specifies  the 
test  conditions  for  acceptance  testing  of  a computer  pro- 
gram. The  test  plan  is  based  on  the  requirements  of  the  FD 
and  the  SS. 

(2)  USERS  MANUAL  (UM) . Describes  in  general,  non- 
technical terms,  the  applications  of  the  system  and  its  com- 
puter programs,  and  provides  specific  information  about 
inputs  and  outputs  to  assist  the  user  in  effectively  uti- 
lizing the  system. 

(3)  COMPUTER  OPERATION  MANUAL  (OM) . Contains  the 
information  necessary  to  operate  the  ADS,  including  deck 
setups,  inputs  required,  operating  instructions,  error  mes- 
sages, and  recovery  procedures. 

(4)  DEVELOPMENT  TEST  PLAN.  A document  that  speci- 
fies method  and  content  for  development  testing.  Defines 
test  management,  reports,  controls,  manpower,  acceptance 
criteria,  and  test  procedures  (Attachment  2) . 

(5)  PROGRAM  MAINTENANCE  MANUAL  (MM) . A document 


If 


maintained  by  a programmer  or  programmer  team  for  each 


module  for  which  responsibility  is  assigned.  It  includes 
requirements,  interfaces,  design,  and  test  information 
(Attachment  3) . 

(6)  OTHER  PLANS.  Detailed  supporting  plans  required 
as  annexes  to  the  Data  Project  Plan  (see  AFR  300-12,  Volume 
I)  are  normally  prepared  during  the  development  phase.  Such 
plans  address  training,  communications,  site  preparation. 


c.  REVIEWS.  The  following  reviews  are  conducted  during 
the  development  phase. 

(1)  PRELIMINARY  DESIGN  REVIEW  (PDR) . The  PDR  is  a 
review  of  the  subsystem  design  approach  for  a CPCI  occurring 
prior  to  the  start  cf  detailed  design  (paragraph  4-5) . 

(2)  CRITICAL  DESIGN  REVIEW  (CDR) . The  CDR  is  a 
review  conducted  on  each  module  or  a set  of  modules  within  a 
CPCI  before  translating  the  detailed  design  to  coded  instruc- 
tions (paragraph  4-6) . 

(3)  CONFIGURATION  AUDITS.  Conduct  functional  con- 
figuration audits  (FCA)  and  physical  configuration  audits 
(PCA)  as  required  to  insure  software  products  and  documen- 
tation comply  with  project  requirements.  These  audit 
functions  are  started  prior  to  the  PVR  and  are  updated  prior 
to  the  SVR  (paragraph  4-7) . 


(4)  PRODUCT  VERIFICATION  REVIEW  (PVR).  The  PVR  is 
a review  of  each  CPCI  to  establish  the  product  baseline  for 
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that  CPCI  and  to  insure  preparation  for  the  Test  Phase  has 
been  completed  (paragraph  4-8) . 

2-5.  TEST  PHASE.  The  installation  and  validation  test  of 
the  system  in  the  operational  environment  by  the  designated 
user  representative  or  by  an  independent  agency. 

a.  VALIDATION  TESTING.  Conduct  two-stage  valida- 
tion testing  of  the  system  by  an  independent  test  group. 

In  the  first  stage,  perform  in-house  validation  testing 
of  the  system  requirements,  correct  problems,  and  update 
documentation.  In  the  second  stage,  rerun  validation  test 

for  the  user,  or  review  successful  in-house  validation  test  i 

runs  with  the  user.  In  addition,  validate  program  specifica- 
tions and  other  relevant  documentation  in  both  stages  (para-  ^ 

graph  6-5) . j 

b.  SYSTEM  VALIDATION  REVIEW  (SVR) . The  SVR  validates 
that  performance  of  a CPCI,  as  determined  through  validation 
testing,  complies  with  the  SS  and  the  FD  (paragraph  4-9) . 

2-6.  OPERATION  PHASE.  The  operation,  maintenance,  and  pro- 
duct improvement  by  the  user  and/or  the  developer. 

2-7.  PROJECT  MANAGEMENT  FUNCTIONS.  The  organizational 
structure  selected  to  develop  an  ADS  project  is  dictated  to 
a certain  degree  by  available  manpower,  the  project's  scope 
and  complexity,  schedule  requirements,  and  other  project- 
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unique  factors.  The  project  manager  is  ultimately  responsible 
for  insuring  adherence  to  management  policies  and  practices. 

He  must  plan  major  functions  of  the  project  and  insure  that 


appropriate  planning  is  being  conducted  at  lower  levels. 
Primary  management  functions  are  briefly  introduced  here, 
and  covered  in  detail  in  later  chapters. 

a.  QUALITY  ASSURANCE.  Quality  assurance 

consists  of  all  actions  that  are  taken  to  assure  the  develop- 
•'  ment  organization  delivers  products  that  meet  performance 

• requirements  and  adhere  to  standards  and  procedures.  (See 

Chapter  5.) 

b.  CONFIGURATION  MANAGEMENT.  A discipline  applying 
technical  and  administrative  direction  and  surveillance  to 
(1)  identify  and  document  the  functional  and  physical  char- 
acteristics of  a configuration  item,  (2)  control  changes  to 
those  characteristics,  and  (3)  record  and  report  change  nro- 
cessing  and  implementation  status.  (See  Chapter  3.) 

c.  TESTING.  Testing  verifies  that  software  products 
conform  to  performance  and  design  requirements.  Test  plans 
and  procedures  must  be  prepared,  test  data  made  available, 
test  facilities  provided,  and  schedules  must  include  adequate 
test  time.  (See  Chapter  6.) 

e.  PLANNING  AND  CONTROL.  The  planning  and  control  func- 
tion incJudes  preparation  of  budget  estimates  and  tracking 
of  actual  costs;  tracking  progress  against  project  schedules; 
monitoring  reviews  and  audits;  and  preparation  of  the  docu- 
mentation plan. 


f.  SOFTWARE  DEVELOPMENT.  Software  development  includes 
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Chapter  3 

CONFIGURATION  MANAGEMENT 

3-1.  PURPOSE.  This  chapter  provides  detailed  procedures 
for  software  configuration  management,  from  the  conceptual 
through  the  operation  phases  of  a system.  It  provides  a 
general  software  configuration  management  approach  which  can 
be  tailored  to  specific  project  requirements.  The  key  to 
configuration  management  is  knowing  what  you  have  and  keeping 
track  of  changes. 

3-2.  CONFIGURATION  MANAGEMENT  PLAN  (CMP) . Software  config- 
uration management  must  be  responsive  to  user  requirements. 

Complex  software  projects  require  a highly  organized  con- 
figuration management  (CM)  activity  to  insure  project  objec- 
tives are  achieved;  whereas,  less  complex  software  projects 
may  require  nothing  more  than  control  of  applicable  specifi- 
cations and  acceptance  testing  of  the  products.  A CMP  is  j 

part  of  the  Data  Project  Plan  and  describes  project  responsi-  ‘ 

j 

bilities  and  procedures  for  implementing  CM,  and  includes  ! 

I 

schedules  for  major  CM  activities  and  events  for  life  cycle 
of  the  system.  The  CMP  should  be  developed  and  approved 
prior  to  start  of  the  definition  phase  (Attachment  4) . 

3-3.  CONFIGURATION  IDENTIFICATION.  Underlying  all  other 
aspects  of  software  configuration  management  is  a progres- 
sive and  increasingly  detailed  identification  of  the  compu- 
ter program  configuration  by  means  of  baseline  documenta- 
tion. The  required  configuration  of  a computer  progreim 
configuration  item  (CPCi)  is  identified  by  its  system/subsystem 


specification  (SS) , and  its  achieved  configuration  by  its 
program  specification  (PS) . Other  documents  that  may  be 
part  of  a CPCI's  configuration  identification  are  the  data 
requirements  document  (RD) , if  it  is  referenced  in  the  SS, 
and  the  data  base  specification  (DS) , if  it  is  referenced  in 
the  PS.  Other  documents  are  not  part  of  a CPCI's  configura- 
tion identification.  They  are  influenced  by,  but  do  not 
identify  or  control  a CPCI's  configuration  identification. 

a.  DESIGNATING  A CPCI.  The  term  CPCI  is  used  to  define 
the  product  or  products  subject  to  the  application  of  con- 
figuration management  procedures.  A CPCI  may  be  an  entire 
ADS  or  portion  thereof.  This  determination  is  basically  a 
management  decision.  Trade-offs  are  based  on  judgement  and 
experience.  Factors  bearing  on  the  decision  include  the 
risk  associated  with  the  ADS,  its  planned  use  (single  or 
multiple  users) , its  complexity,  the  level  of  change  control 
required  by  management,  the  modification  activity  expected 
during  the  life  of  the  ADS,  and  cost  thresholds  imposed  by 
management  based  on  the  estimated  value  of  the  ADS.  For 
example,  an  ADS  may  consist  of  a data  base  management  system 
and  several  application  programs.  The  ADS  may  be  defined  as 
a single  CPCI  or  each  logical  component  may  be  defined  as  a 
CPCI. 

b.  SELECTION  OF  PROJECT  DOCUMENTS.  To  determine  the 
items  needed  in  a project  documentation  set,  a documentation 
analysis  should  be  performed  during  the  definition  phase. 
Such  an  analysis  will  help  to  insure  that  project  documenta- 
tion adequately  covers  all  information  needed  with  a minimum 


of  overlap  or  duplication. 
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c.  PREPARING  PROJECT  DOCUMENTATION . The  development 
activity  is  normally  responsible  for  preparing  and  deliver- 
ing the  required  documentation.  Documentation  responsibilities 
should  be  prepared  specifying  the  individuals  that  are 
responsible  for  preparing  and  coordinating  each  document 
from  origination  to  formal  release  and  updating. 

(1)  Each  project  manager  should  insure  that! 

(a)  Documentation  requirements  are  adequately 
and  appropriately  identified  for  the  project. 

(b)  A documentation  plan  is  established  as  early 
as  possible  in  the  ADS  life  cycle. 

(c)  Project  documentation  requirements  are 
understood  by  personnel  responsible  for  documentation  prepa- 
ration, coordination,  distribution  and  updating. 

(2)  The  personnel  preparing  each  document  should  be 
responsible  for: 

(a)  Thoroughly  understanding  the  project  docu- 
mentation requirements  as  specified  in  the  documentation 
plan. 

(b)  Understanding  the  intended  use  of  the  docu- 
ment and  the  requirements  of  the  intended  users. 

(c)  Understanding  the  overall  content  and  appro- 
priate level  of  detail  required  for  the  document. 

(d)  Complying  with  applicable  ADP  standards 
for  document  format  and  content. 


d.  PREPARING  A SOFTWARE  DOCUMENTATION  PLAN.  A documen-  ! 

tation  plan  is  first  developed  in  draft  form  and  usually 

i 

refined  during  the  definition  phase  of  the  system.  Essential  l 

steps  in  preparing  a software  documentation  plan  are: 

(1)  Establish  the  scope  and  general  outline  of  the  . j 

I 

documentation  plan.  Generally  the  documentation  plan  covers 
technical  documentation,  management  documentation,  reports 
and  official  letters,  and  standard  forms  to  be  used  on  the 
project. 

(2)  Define  the  specific  document  types  that  have 
been  selected  for  the  project.  For  each  such  document  state: 

(a)  Title  and  identification  number. 

(b)  Purpose. 

(c)  Responsibilities  and  general  procedure  for 
organization  and  updating. 

(d)  Coordination  and  approval  authorities. 

(3)  Schedule  each  document  to  be  available  at  the 
time  its  information  is  required  for  review,  or  release, 
based  on  the  projects  particular  need. 

» 

M)  Define  responsibilities  and  procedures  for  pub-  ! 

X.  i.  j 

lication  and  distribution  of  documents. 

e . DOCUMENTATION  STANDARDS . 

(1)  Documentation  for  ADS  development  will  be  pre- 
pared in  accordance  with  DOD  Manual  4120. 17-M,  AFR  300-12, 

Volume  I,  and  this  regulation. 
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(2)  Technical  documentation  may  be  combined  under 


one  cover  with  each  type  designated  as  a separate  part; 
e.g..  Part  I,  Part  II,  etc.,  and  the  format  standard  for 
each  type  still  followed. 

f.  COMPUTER  PROGRAM  LIBRARIES.  Computer  program  libraries, 
also  called  progrcimming  support  libraries,  are  automated  data 
repositories  that  have  been  successfully  used  as  an  aid  in  M)S 
development  projects  to  the  actual  software  development  process 
and  the  management  of  the  project.  Project  managers  should 
consider  the  use  computer  program  labraries  early  in  the  defini- 
tion phase. 

(1)  Support  of  the  software  development  process 
includes; 

(a)  Storage  and  maintenance  of  computer  programs 
and  related  data. 

(b)  Compilation  and  testing  of  computer  programs. 

(c)  Generation  of  documentation. 

(2)  Support  to  project  management  includes: 

(a)  Collection  and  reporting  of  management  data 
related  to  software  development. 

(b)  Control  over  the  integrity  and  security  of  data 
stored  in  the  library. 

3-4.  CONFIGURATION  CONTROL.  Configuration  control  is  the 
systematic  evaluation,  coordination,  approval  or  disap- 
proval, and  implementation  of  all  approved  changes  in  the 


i 
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configuration  of  a CPCI  after  formal  establishment  of  its 
configuration  identification.  This  process  assures  adequate 
analysis  of  changes  that  affect  cost,  schedule  or  design, 
and  provides  controlled  updating  of  evolving  software  con- 
figuration throughout  the  ADS  life  cycle.  This  paragraph 
describes  the  concepts,  forms,  and  procedures  for  achieving 
the  proper  degree  of  configuration  control  during  all  phases 
of  a project. 

a.  CHANGE  CLASSIFICATIONS.  Changes  to  software  pro- 
ducts or  documentation  can  be  placed  into  three  classifica- 
tions each  requiring  different  processing  and  review  proce- 
dures depending  on  the  impact  of  the  change. 

(1)  CLASS  I CHANGES.  Class  I changes  are  changes 
which  impact  approved  schedules,  cost,  or  baselines.  Class 
I changes  require  approval  by  the  configuration  control 
board  (CCB)  and/or  the  appropriate  ADP  approval  authority  if 
the  change  exceeds  the  authority  delegated  to  the  project 
manager.  Generally  they  include  the  following  kinds  of 
changes ; 

(a)  Technical  changes  to  the  functional,  allo- 
cated, or  product  baselines  of  the  CPCI. 

(b)  Changes  involving  performance  outside  the 
specified  tolerances  of  the  CPCI  or  affecting  interface 
characteristics  with  other  CPCI’s. 

(2)  CLASS  II  CHANGES.  Class  II  changes  are  changes 
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affecting  documentation  only,  or  changes  which  are  classified 


as  maintenance.  Class  II  changes  do  not  require  configura- 
tion control  board  (CCB)  approval  prior  to  implementation, 
but  copies  of  Class  II  changes  must  be  forwarded  to  the 
project  manager  (or  his  designee)  for  concurrence  in  the 
change  classification. 

(3)  INTERNAL  CHANGES.  Internal  changes  are  changes 
to  a CPCI,  or  document  that  has  not  been  released  for  Class 
I and  Class  II  change  control.  For  example,  changes  to  a 
preliminary  program  specification  (PS)  prior  to  the  product 
baseline  are  treated  as  internal  changes.  Similarly,  any 
changes  required  during  development  testing  are  treated  as 
internal  changes  unless  the  change  affects  the  functional  or 
allocated  baselines,  costs,  or  schedules. 


CONFIGURATION  CHANGE  CONTROL  PHASING  DURING  PROJECT  LIFETIME 


Figure 


b.  CHANGE  CONTROL  PHASING.  Configuration  change  con- 
trol tasks  and  phasing  discussed  in  this  section  are  con- 


I h- 


I 

cerned  primarily  with  the  development  phase.  Change  control 
during  other  phases  is  also  briefly  considered.  A general 
view  of  when  change  control  is  applied  is  shown  in  Fig.  3-1. 

(1)  When  a functional  description  (FD)  and  a system/ 
subsystem  specification  (SS)  have  been  baselined  at  the 
start  of  development,  they  are  under  Class  I and  Class  II 
change  control  throughout  the  ADS  life  cycle.  This  level  of 
control  also  applies  to  the  data  requirements  document  (RD) 

and  the  data  base  specifications  (DS)  if  they  are  separate  docu- 
ments . 

(2)  Internal  change  control  of  coded  modules  first 
occurs  when  modules  are  released  for  development  testing. 

Following  establishment  of  the  product  baseline,  the  CPCI 
and  the  CPCI  progrcim  specification  are  placed  under  Class  I 
and  II  change  control  and  continue  at  this  control  level 
throughout  the  ADS  life  cycle. 

(3)  The  test  plan  (PT)  is  usually  placed  under  Class 
I and  Class  II  change  control  about  30  days  before  delivery 
of  the  product  for  Environment  System  Test,  Phase  I (EST  I) . 

The  computer  operation  manual  (OM)  and  program  maintenance 
manual  (MM)  are  placed  under  class  I and  Class  II  change 
control  at  the  product  baseline. 

c.  CONFIGURATION  CONTROL  BOARD  (CCB) . Configuration 
control  boards  are  organized  to  review,  evaluate,  approve  or 


disapprove,  and  release  all  Class  I changes.  As  previously 
indicated.  Class  I changes  may  require  approval  of  both  the 
CCB  and  the  appropriate  ADP  approval  authority.  The  CCB 
must  approve  Class  I changes  before  they  are  submitted  (if 
required)  to  the  appropriate  ADP  approval  authority. 

(1)  The  CCB  usually  is  chaired  by  the  project  man- 
ager. The  members  are  the  user  representative,  development 
and  quality  assurance  personnel.  Attendance  at  meetings 
depends  on  impact  of  the  change  request  being  reviewed. 

(2)  The  CCB  is  organized  at  the  start  of  the  project 
Policies  and  procedures  required  for  the  operation  of  the 
CCB  will  be  prepared  and  implemented.  A secretary  will  be 
appointed  with  responsibility  for  preparing  the  agenda, 
agenda  items,  and  meeting  minutes. 

d.  CHANGE  CONTROL  FORMS.  Change  control  forms  are  the 
basic  media  for  reporting  problems,  and  processing  changes. 
They  are  the  key  source  of  information  concerning  the  status 
of  changes  during  change  processing.  Changes  to  baseline 
documentation  must  be  controlled  using  the  Baseline  Change 
Request  (BCR),  or  equivalent  form,  paragraph  d(3).  With  the 
exception  of  the  BCR,  use  of  the  following  forms  is  optional 
Sample  forms  and  instructions  for  preparation  are  contained 
in  Attachment  6 . 

(1)  SOFTWARE  PROBLEM  REPORT  (SPR) . An  SPR  is  used 
after  initiation  of  development  testing  to  report  a known  or 
suspected  discrepancy  or  deficiency  in  an  existing  computer 


program,  in  its  documentation,  or  in  interfacing  software. 

The  SPR  provides  a record  of  the  discrepancy  from  initiation 
to  problem  resolution. 

(2)  DATA  BASE  CHANGE  REQUEST  (DBCR) . A DBCR  is 
used  to  control  changes  to  a data  base  after  the  data  base 

V in  placed  under  configuration  control.  When  a problem  is 

reported  via  an  SPR  and  a solution  to  the  problem  requires  a 
data  base  change,  a DBCR  is  prepared  and  distributed.  'After 

» 

. approval  of  the  DBCR,  the  data  base  is  updated  accordingly. 

(3)  BASELINE  CHANGE  REQUEST  (BCR) . A BCR  or  equivalent 
form  is  used  to  propose  Class  I and  Class  II  changes.  It 
provides  a description  of  the  proposed  change  and  the  impact 

of  the  change  on  the  cost,  schedule,  and  performance  of  the 
CPC  I. 

(4)  DESIGN  PROBLEM  REPORT  (DPR) . The  DPR  provides  a 
standard  means  for  documenting  problems  identified  during 
formal  reviews  and  audits. 

(5)  REQUEST  FOR  DEVIATION/WAIVER.  A request  for 
deviation  or  waiver  is  used  by  a developer  to  request  and 
document  temporary  departures  from  configuration  identifica- 
tion requirements  when  permanent  changes  are  not  acceptable. 

A deviation  is  a written  authorization  granted  prior  to  pro- 
duct development  to  permit  a developer  to  depart  from  a par- 
ticular performance  or  design  requirement  for  a specified 
product  or  period  of  time.  A waiver  is  a written  authoriza- 
tion to  deliver  a configuration  item  that  has  been  found 
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after  devolopment  to  depart  from  specified  requirements  but 
that  nevertheless  is  considered  suitable  for  use  or  rework. 

A BCR  may  result  in  a deviation  or  waiver. 

e.  CHANGE  PROCESSING.  A change  may  be  requested  by 
any  organization  associated  with  the  project.  All  change 
requests  initiated  in  a project  are  first  reviewed  at  the 
lowest  approval  level  in  the  project.  If  the  change  does 
not  affect  approved  baselines,  action  should  be  implemented 
at  the  lowest  level.  Change  processing  guidelines  are: 

(1)  CLASS  I CHANGE  CONTROL.  Class  I changes  require 
a BCR  to  be  submitted  for  approval  to  the  configuration 
control  board  (CCB) . If  the  change  exceeds  delegated  approval 
thresholds,  the  BCR  is  used  to  prepare  an  amended  DAR  which 

is  submitted  to  the  appropriate  ADP  approval  authority. 
Instructions  for  submitting  an  amended  DAR  are  contained  in 
AFR  300-12,  Volume  I.  Implementation  of  most  Class  I 
changes  follows  an  independent  development  cycle  until  it 
can  be  merged  with  the  current  software  system.  After  start 
of  the  operation  phase,  all  modifications  are  processed  as 
DARs . 

(2)  CLASS  II  CHANGE  CONTROL. 

(a)  Prior  to  the  operation  phase.  Class  II 
changes  must  be  approved  at  the  level  designated  by  the 
project  manager.  Concurrence  of  proper  classification  should 
be  obtained  from  the  project  manager  (or  his  designee) . 

BtSl  AVAILABLE  COPV 


, 

i 


I 


j 


5/ 


(b)  During  the  operation  phase.  Class  II  changes 


r 

I 


t 


p 


must  be  approved  by  the  ADS  manager  or  his  designee. 

(c)  Release  of  a Class  II  change  is  authoriza- 
tion to  implement  the  change. 

(3)  INTERNAL  CHANGE  CONTROL. 

(a)  Internal  changes  do  not  require  CCB  approval 
or  concurrence. 

(b)  Modules  verified  or  updated  during  development 
testing  are  included  in  the  project  library  and  integrated  into 
the  CPCI.  In  certain  real-time  and  on-line  applications  pro- 
grams may  be  patched  to  conserve  limited  test  time  and  main- 
tain development  schedules.  All  "patches"  must  be  dociimented 
in  a log  and  entered  as  changes  into  the  CM  system. 


f.  MULTIVERSION  CPCIs 

(1)  More  than  one  version  of  a CPCI  may  exist  with 
each  version  in  a different  phase  of  the  ADS  1? fe  cycle. 
Multiple  versions  can  occur  under  the  following  circumstances. 

(a)  Follow-on  development  to  an  AJ)S  currently  in 
its  operation  phase.  Changes  which  are  not  of  an  emergency 
nature  should  be  collected  and  implemented  as  a single 
release  when  feasible.  They  are  usualy  referred  to  as 
"block  releases". 
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of  a product  currently  at  the  product  baseline,  with  the  new 
version  to  be  started  at  the  allocated  baseline. 
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(d)  A product  being  developed  incrementally 
to  utilize  manpower  at  a steady  rate. 

(2)  A major  consideration  in  controlling  multiple 
versions  is  the  coordination  of  changes.  For  example, 
version  "A",  currently  in  system  testing,  has  functional, 
allocated,  and  product  baselines,  while  version  "B",  in 
development  testing,  has  only  functional  and  allocated 
baselines.  A change  of  small  consequence  to  "B"  might  have 
severe  implications  on  "A".  Therefore,  all  versions  must  be 
considered  whenever  a change  is  incorporated  in  any  of  them. 
To  insure  that  such  consideration  has  been  given,  changes 
should  be  logged  against  all  effected  versions. 


g.  MULTIPLE  CONFIGURATION  AND  MULTIPLE  SITE  PROJECTS . 
(1)  Multiple  configurations  may  also  exist  and  be 
controlled  at  the  same  time.  This  situation  occurs  when  an 
ADS  is  installed  or  about  to  be  installed  at  different 
locations,  and  each  location  is  to  maintain  its  own  configu- 


ration. Each  location's  ADS  is  treated  as  a separate  CPCI, 
and  problems  at  any  location  should  be  submitted  to  other 
locations  to  see  if  there  is  any  impact. 

(2)  When  a single  configuration  is  required  for  all 
locations,  the  configuration  is  controlled  from  one  loca- 
tion. The  other  locations  maintain  a record  of  any  temp- 
orary deviations  from  the  configuration. 
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h.  SMALL  PROJECTS.  On  projects  less  than  one  man-year  of 


1 


development  effort  where  constant  informal  contact  among  all 
participants  is  possible,  there  may  be  no  need  for  formal 
change  control  procedures.  However,  program  maintenance  manuals 
should  be  maintained  to  document  the  development  process, 
including  problems,  corrective  action,  and  current  configurations. 
3-5.  CONFIGURATION  STATUS  ACCOUNTING,  Configuration  status 
accounting  is  the  process  of  recording  and  reporting  the 
status  of  an  evolving  CPCI  throughout  the  ADS  life  cycle. 

Status  accounting  records  include  a log  for  each  change 
control  form,  with  sufficient  cross  referencing  between  logs 
to  facilitate  tracking  and  preparation  of  reports.  The 
logs  may  also  be  employed  for  analyses  of  development  and 
test  activities.  Attachment  5 contains  a brief  description 
of  logs  which  may  be  used  for  configuration  status  account- 
ing. The  following  documents  may  be  used  to  report  infor- 
mation regarding  the  ADS. 

a.  TURN-OVER  (TRANSMITTAL)  LETTER,  The  turn-over 
(transmittal)  letter  identifies  all  products  delivered  and 
cross-references  applicable  documentation.  The  turn  over 
letter  contains  the  following  information  as  a minimum: 

(1)  INVENTORY  OF  MATERIALS  RELEASED.  A list  of 
all  items  (tapes,  cards,  disks,  etc.)  forv^arded  with  the 


release.  All  utility  and/or  support  computer  programs, 
not  part  of  the  release,  but  required  to  operate,  load,  or 
regenerate  the  release  are  identified. 
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(2)  ADAPTATION  DATA.  For  the  release  of  a new 
CPCI  version,  identify  (by  reference  to  appropriate  spe- 
cifications and/or  listings)  all  unique-to-site  data  in  the 
items  being  released. 

(3)  INTERFACE  COMPATIBILITY.  For  the  release  of 
a new  CPCI  version,  indicate  other  systems  and/or  CPCIs 
affected  by  the  changes  incorporated  in  this  release. 

(4)  BIBLIOGRAPHY  OF  REFERENCE  DOCU’iENTS.  This 
section  is  included  only  for  the  release  of  a new  version  of 
a CPCI  and  lists  all  pertinent  documents  related  to  the 
released  version. 

(5)  INSTALLATION  INSTRUCTIONS.  Describe  (either 
directly  or  by  reference)  the  method  used  to  install  and 
check  out  the  delivered  CPCI  version  or  change. 

(6)  PROBLEMS  AND  RESTRICTIONS.  List  all 
known  problems  associated  with  the  delivered  CPCI  and  any 
restrictions  pertaining  to  its  use. 

b.  PRODUCT  STATUS  REPORTS. 

C.  COMPUTER  PROGRAM  CONFIGURATION  ITEM  (CPCI)  INDEX. 

(1)  A CPCI  index  describes  the  current  status  of 
all  baseline  specifications  and  support  documents  pertaining 
to  a CPCI.  Document  status  is  given  in  terms  of  issue 
dates,  document  numbers  and  titles,  BCRs  and  revision 
identifiers  associated  with  each  document  issue  or  revision 
resulting  from  implemented  changes.  The  index  also  contains 
a development  record  that  gives  completion  dates  for  the  key 


events  of  CPCI  development. 

(2)  A CPCI  index  is  initially  issued  within  30 
days  of  the  establishment  of  the  allocated  baseline  for 
each  CPCI  and  is  updated  as  required.  Each  succeeding  issue 
is  expanded  and  revised  to  reflect  progress  through  the  key 
events. 
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Chapter  4 

REVIEWS  AND  AUDITS 

4-1.  PURPOSE.  To  describe  the  reviews  and  audits  that  are 
scheduled  at  meaningful  points  during  the  ADS  life  cycle. 
4-2.  PARTICIPATION  AND  RESPONSIBILITIES.  The  project 
manager  or  his  designee  chairs  all  reviews  and  audits.  The 
user  representative  will  make  sure  that  the  functional 
requirements  are  being  adequately  met. 

a.  Project  Manager  Responsibilities. 

(1)  Establish  the  time,  the  place,  and  the  agenda 
for  the  reviews  and  audits,  and  coordinate  these  arrange- 
ments with  all  participants.  Depending  on  the  type  of 
review  or  audit,  schedules  for  incremental  reviews  may  have 
to  be  prepared  and  coordinated  with  all  participants. 

(2)  Prepare  for  each  review  or  audit  in  sufficient 
depth  consistent  with  project  scope  and  size.  Material  to 
be  reviewed  must  be  submitted  to  participants  prior  to  each 
review,  allowing  time  for  comment.  Documents  should  be 
reviewed  for  technical  rather  than  editorial  content. 

Review  participants  should  prepare  design  problem  reports 
(DPRs)  and  submit  them  to  the  originator  of  the  material 
being  reviewed.  The  originators  of  the  material  will  evalu- 
ate the  DPRs  and  reply  to  them  at  the  formal  review. 

Another  approach  is  to  correct  minor  discrepancies  prior  to 
the  review,  advise  participants  of  the  disposition,  and  hold 
the  major  DPRs  for  discussion  at  the  meeting.  After  the 
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review,  the  documentation  is  updated,  incorporating  necessary 
changes,  and  formally  released. 
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(4)  Provide  a secretary  to  record  official  meeting 
minutes.  Minutes  are  recorded  only  as  directed  by  the 
chairman  or  the  user  representative  and  consist  of  signifi- 
cant questions  and  answers,  action  items,  deviations,  con- 
clusions, and  recommended  courses  of  action  resulting  from 
presentations  or  discussions. 

(5)  Publish  and  distribute  official  minutes  of  the 

meeting. 

(6)  After  each  review  or  audit,  provide  the  review 
participants  with  the  development  activity's  position  on 
each  action  item  in  the  minutes. 

b.  User  Responsibilities. 

(1)  Coordinate  the  arrangements  for  the  review  or 
audit  with  the  project  manager. 

(2)  Review  all  documentation  provided  by  the 
development  activity  prior  to  the  review  or  audit  to  vali- 
date that  user  requirements  are  stated  properly  and  have 
been  adequately  translated  into  the  ADS  design.  Discrepan- 
cies should  be  documented  by  DPR  and  submitted  to  the  devel- 
opment activity. 

(3)  Within  30  days  following  each  review  or  audit, 
provide  review  participants  with  a position  on  each  action 
item  in  the  minutes. 


c.  Resolution  of  disagreements  between  the  development 
activity  and  the  user  will  be  escalated  to  the  next  level  of 
management  for  resolution. 

4 4-3.  SYSTEM/SUBSYSTEM  REQUIREMENTS  REVIEW  (SRR) . The  SRR 

is  conducted  during  the  conceptual  phase  to  assure  the  user 
that  the  project's  progress  is  responsive  to  the  approved 
requirements,  and  to  determine  initial  direction  of  the 
design  effort. 

a.  Review  Items.  The  principal  items  to  be  reviewed  at 

• the  SRR  are  the  Data  Automation  Requirement  (DAR) , the  Data 

Project  Directive  (DPD) , the  Data  Project  Plan  (DPP) , and 
the  draft  Functional  Description  (FD) . A detailed  review  of 
mission  and  operational  requirements,  preliminary  require- 
ments allocation,  system  interface  studies,  etc. , should  be 
made  at  this  time. 

b.  Postreview  Action.  After  completion  of  the  SRR,  the 
minutes  are  distributed  by  the  project  manager.  Notifi- 
cation of  any  required  postreview  action  items,  plus  offi- 
cial acknowledgment  of  conduct  and  completion  of  the  review, 
are  provided  to  the  user,  the  developer,  and  the  appropriate 
ADP  approval  authority.  Approval  of  the  FD  at  the  SRR 
establishes  the  Functional  Baseline  and  initiates  configura- 
tion management  of  the  FD. 

4-4.  SYSTEM  DESIGN  REVIEW  (SDR).  The  SDR  establishes  the 
allocated  baseline  for  individual  CPCIs.  Upon  approval  of 


the  system/subsystem  specification  (SS) , configuration 
management  of  the  SS  is  initiated. 

a.  Review  Items. 

( 1 ) The  SDR  documentation  package  should  include 
the  following: 

(a)  Functional  Description  (FD)  with  changes. 

(b)  System/subsystem  specifications  (SS) . 

(c)  Tradeoff  studies,  analyses,  etc. 

(d)  Data  Project  Plan  (DPP) . 

(2)  The  SDR  is  conducted  to  accomplish: 

(a)  Review  any  changes  to  the  FD. 

(b)  Insure  the  allocated  requirements  implement 
the  functional  requirement  stated  in  the  FD. 

(c)  Insure  the  technical  project  risks  are 
identified,  ranked,  avoided,  and  adequate  tradeoffs  are 
made. 

(d)  Insure  a technical  understanding  of 
requirements  has  been  reached  and  technical  direction  is 
provided  to  project  participants. 

(e)  Insure  project  schedules  and  milestones 
are  consistent  with  the  allocated  requirements. 

b.  Postreview  Action.  After  completion  of  the  SDR,  the 
minutes  of  the  meeting  are  coordinated  with  all  partici- 
pants. Participants  will  also  be  advised  of  any  postreview 
action  items  requiring  response.  Approval  of  the  the  SS 
establishes  the  allocated  baseline. 


4-5.  PRELIMINARY  DESIGN  REVIEW  (PDR) . The  PDR  is  a technical 
review  of  the  subsystem  design  for  a computer  program  con- 
figuration item  (CPCI) . The  PDR  for  each  CPCI  occurs 
between  SDR  and  CDR  in  the  development  phase.  Only  one 
successful  PDR  per  CPCI  is  required.  A collective  PDR  for  a 
functionally  related  group  of  CPCIs,  treating  each  CPCI 
individually,  may  be  held  when  advantageous  to  the  project. 

The  overall  technical  risks  associated  with  each  CPCI  are 
also  reviewed  on  a technical,  cost,  and  schedule  basis. 

NOTE:  If  the  ADS  is  defined  as  a single  CPCI,  the  PDR  may 

be  combined  with  the  SDR. 
a.  Review  Items. 

(1)  The  PDR  documentation  package  should  include 
the  following. 

(a)  Updated  functional  description  (FD) . 

(b)  System/subsystem  specification  (SS) , 
including  approved  changes. 

(c)  Draft  of  CPCI  test  plans  (PT) , less  test 

procedures . 

id)  Data  base  structure  and  orgaai zat Lon . 

(e)  Design  problem  reports  (DPR) . 

(2)  The  PDR  is  conducted  to  establish  the  following: 

(a)  Compatibility  of  the  design  with  the  SS 
for  the  CPCI  and  with  other  CPCIs  with  which  it  must 

(b)  Adequacy  of  test  plan. 
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interface. 


(3)  To  satisfy  the  intent  of  the  PDR,  the  following 


w 
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tasks  should  be  performed. 

(a)  Review  all  functional  interfaces.  Review 
message  formats,  storage  availability,  and  other  considera- 
tions that  may  have  been  established  in  the  SS.  At  this 
time,  the  interfaces  between  CPCI  and  hardware  CIs  should  be 
defined  at  a level  low  enough  to  satisfy  all  software 
development  needs. 

(b)  Review  the  structure  of  the  CPCI  as  a 
whole,  with  emphasis  on  allocation  of  the  functions  deline- 
ated in  the  SS  to  individual  computer  programs;  storage 
requirements  and  allocation;  computer  program  operating 
sequences;  and  design  of  the  data  base. 

(c)  Analyze  critical  timing  requirements  to 

insure  the  proposed  CPCI  design  satisfies  the  timing  requirements.. 

(d)  Review  human  factors  impact  of  proposed 
subsystem  design  of  the  CPCI. 

(e)  Review  all  changes  to  the  SS  subsequent  to 
the  established  allocated  baseline.  Insure  the  FD  and  SS 
adequately  reflect  these  changes. 

(f)  Review  the  PT  to  insure  it  satisfies  the 
requirements  of  the  FD  and  SS. 

(g)  Review  status  of  all  negative  and  provisional 
entries  such  as  "not  applicable"  or  "to  be  determined"  in 

the  SS. 


b.  Postreview  Action.  After  completion  of  the  PDR,  the 


i 

I 

I 

i 

I 

1 


minutes  are  published  and  distributed  by  the  development 
activity.  The  user  will  be  notified  of  any  required  post- 
review action. 


4-6.  CRITICAL  DESIGN  REVIEW  (CDR) . 

a.  Purpose.  The  CDR  is  a review  conducted  during  the 
development  phase  before  translating  logic,  and  algorithms 
to  coded  instructions.  The  CDR  insures  the  detailed  design 
solution,  as  reflected  in  the  draft  program  specification 
(PS) , satisfies  performance  requirements  established  by  the 
SS.  The  CDR  also  verifies  system  design  compatibility  with 
other  CPCIs,  and  within  the  CPCI.  Approval  of  the  project 
manager  is  required  to  release  portions  of  CPCIs  for  coding 
prior  to  CDR  when  necessary  to  maintain  schedules. 

b.  Review  Items  and  Tasks 

(1)  The  CDR  documentation  should  consist  of  the 


following: 

(a)  System/subsystem  specification  (SS) , 
including  approved  changes. 

(b)  CPCT  test  plans  (PT) , including  test 

procedures. 

(c)  Design  problem  reports  (DPR) . 

(d)  Preliminary  design  review  (PDR)  minutes. 

(e)  Draft  of  computer  operation  manual  (OM) . 

(f ) Draft  of  program  maintenance  manual  (MM) . 

(g)  Draft  of  the  program  specification  (PS) . 
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(h)  Draft  of  users  manual  (UM) . 

(i)  Appropriate  support  documentation,  such  as 
updated  timing  studies,  accuracy  studies,  etc. 

(2)  As  a minimum,  the  following  tasks  are  to  be 
performed: 

(a)  Determine  compatibility  of  the  PS  with  the 


SS. 


(b)  Review  all  interfaces  between  CPCIs  and 
within  a CPCI . 

(c)  Review  interfaces  between  CPCI  and  the 
applicable  hardware  CIs  to  insure  changes  have  not  affected 
compatibility. 

(d)  Review  the  PT  for  technical  adequacy  and 
compatibility  with  the  FD  and  SS  requirements. 

c.  Postreview  Action.  After  completion  of  the  CDR,  the 
minutes  are  published  and  distributed.  Participants  will  be 
notified  of  any  required  postreview  action.  Successful  com- 
pletion of  the  CDR  permits  commencement  of  programming. 

4-7.  CONFIGURATION  AUDITS. 

a.  Purpose.  A configuration  audit  is  a process  employed 
to  verify  conformance  of  a CPCI  to  specifications  and  stand- 
ards. Two  audits,  the  Functional  Configuration  Audit  (FCA) 
and  the  Physical  Configuration  Audit  (PCA) , are  required  at 
two  points  during  the  ADS  life  cycle  for  an  ADS  requiring 
greater  than  fifty  development  manyears.  The  first  point, 
prior  to  the  Product  Verification  Review  (PVR) , is  termed 
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the  preliminary  FCA  and  PCA,  and  the  second  point  is  prior 
to  the  Sy-.tem  Validation  Review  (SVR)  and  termed  the  final 
FCA  and  PCA.  The  preliminary  audits  are  a thorough  tech- 
nical examination  of  the  CPCI,  while  the  final  audits  are 
directed  at  changes  made  to  the  CPCI  as  a result  of  formal 
testing.  The  following  guidelines  for  conducting  the  FCA 
and  PCA  are  provided. 

b.  FUNCTIONAL  CONFIGURATION  AUDIT  (FCA) . The  FCA 
validates  the  CPCIs  actual  performance  compliance  with  the 
SS.  Test  data  is  reviewed  to  validate  that  the  CPCI  per- 
forms as  required.  The  project  manager  provides  two  FCA 
data  packages  to  the  organization  performing  the  FCA,  one  to 
be  delivered  at  some  negotiated  lead  time  prior  to  FCA,  and 
the  other  to  be  made  available  during  the  FCA. 

(1)  Items  to  be  provided  before  audit  are: 

(a)  Nomenclature  (name  or  descriptive  title  of 

the  CPCI) . 

(b)  Identification  of  necessary  documentation. 

(2)  The  documentation  package  to  be  made  available 
for  FCA  consists  of: 

(a)  Program  specification  (PS) , including 


listings. 

(b)  System/subsystem  specification  (SS)  with 
changes,  if  applicable. 

(c)  Test  plan  (PT)  and  when  available,  test 


I 

i 


1 

< 


i 


■ , i 


I 


reports  (RT) . 


(e)  Status  accounting  records  of  SPRs,  DBCRs 

and  BCRs. 

(3)  The  FCA  consists  of  examining  and  evaluating: 

(a)  Test  procedures  and  results  to  determine 
CPCI  conformance  with  the  SS  and  FD  and  quality  assurance 


provisions. 

(b)  Adequacy  of  analysis  or  simulations  where 
performance  parameters  cannot  completely  be  verified  by 
test. 

(c)  Any  requirements  stated  in  the  FD  that 
could  not  be  met,  and  the  developer's  proposed  solution. 

(d)  All  approved  BCRs  to  insure  they  are 
incorporated  and  verified  during  testing. 

(4)  Deficiencies  noted  and  recommended  corrective 
actions  are  documented  in  the  FCA  minutes. 

c.  PHYSICAL  CONFIGURATION  AUDIT  (PCA) . The  PCA  is  a 
formal  examination  of  the  coded  version  of  a CPCI  against 
its  technical  documentation.  The  PCA  includes  a selective 
audit  of  the  program  specification  (PS) , including  logic 
charts,  listings,  and  design  narrative.  It  also  includes  a 
review  of  the  format  and  completeness  of  any  other  documen- 
tation due  for  acceptance.  The  project  manager  provides  two 
PCA  data  packages  to  the  organization  performing  the  PCA  - 
one  to  be  delivered  at  some  negotiated  lead  time  prior  to 
PCA,  and  the  other  to  be  made  available  during  PCA. 

^(o 


Items  to  be  delivered  prior  to  audit  consist 

(a)  PCA  date  and  location. 

(b)  Identification  of  CPCIs  to  be  accepted. 

The  documentation  package  to  be  provided  and 

; made  available  for  PCA  consists  of: 

(a)  System/subsystem  specification  (SS) . 

(b)  Program  specification  (PS)  . 

t 

(c)  Test  procedures  (PT)  and  test  report  (RT) . 

(d)  Draft  computer  operation  manual  (OM) . 

(e)  Draft  program  maintenance  manual  (MM) . 

(f)  CPCI  tape  and  listings. 

(g)  Applicable  configuration  management 

records. 

c.  The  PCA  consists  of  reviewing: 

r 

I (1)  PS  for  format  and  completeness. 

I (2)  Selectively  compare  the  listings  with  documentation, 

f (3)  Computer  operation  manual  (OM)  and  program 

I maintenance  manual  (MM)  for  format  and  completeness. 

' (4)  The  release  and  change  control  system,  to  be 

sure  it  can  properly  control  the  processing  and  formal 
release  of  changes. 

( 4-8  . PRODUCT  VERIFICATION  REVIEW  (PVR) . 

a.  Purpose.  PVR  is  conducted  for  each  CPCI  at  the  end 
of  the  development  phase  to  establish  the  Product  Baseline 
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(2) 


’*k 

r. 


for  that  CPCI  and  to  insure  prepai ation  for  the  test  phase 
has  been  comoleted. 


b.  A preliminary  functional  configuration  audit  (FCA) 
and  a preliminary  physical  configuration  audit  (PCA)  will  be 
conducted  prior  to  the  PVR  to  provide  management  with  an 
independent  technical  assessment  of  the  CPCI. 

c.  Items  reviewed  at  the  PVR  include: 

I * (1)  Findings  and  recommendations  resulting  from  the 

I . preliminary  FCA  and  PCA. 

^ (2)  Test  plans. 

(3)  Preparations  and  schedules  for  the  test  phase. 

d.  Discrepancies  which  would  lead  to  establishing  an 
invalid  product  baseline  and  premature  testing  must  be 
corrected  prior  to  successful  completion  of  the  PVR. 
Completion  of  the  PVR  establishes  the  CPCI  product  baseline 
and  initiates  the  formal  test  phase  as  described  in  Chapter 
6. 

4-9.  SYSTEM  VALIDATION  REVIEW  (SVR) . The  purpose  of  the 
SVR  is  to  review  the  results  of  validation  testing  to  insure 
that  the  ADS  satisfies  the  requir  ■■m.ents  of  the  SS  and  the 
FD.  The  SVR  is  supported  by  the  final  FCA  and  PCA. 

a.  Review  items.  The  following  items  are  reviewed  at 
the  SVR. 

(1)  PVR  minutes. 

(2)  Findings  and  recommendations  resulting  from  the 
final  FCA  and  PCA. 


(3)  Test  Report  (RT)  . 

(4)  Technical  documentation  as  required. 

(5)  Configuration  management  records  as  required. 


b.  A prerequisite  for  successful  completion  of  the  SVR 
is  the  certification  by  the  functional  OPR  or  designated 
functional  representative  that  the  ADS  adequately  satisfies 
the  requirements  stated  in  the  FD.  Completion  of  the  SVR 
terminates  the  test  phase  and  initiates  the  operation  phase. 


Chapter  5 
QUALITY  ASSURANCE 


W 5-1.  PURPOSE.  To  describe  techniques  to  improve  quality,  j 

V i 

reliability,  and  maintainability  of  an  ADS. 

5-2.  DEFINITION.  Quality  Assurance  (QA)  is  all  actions 
taken  to  make  sure  the  development  organization  delivers 
products  that  meet  performance  requirements  and  adheres  to 
standards  and  procedures. 

5-3.  FUNCTION.  The  QA  function  should  be  independent  of 
‘ the  designers  and  programmers.  This  function  may  be  performed 

by  an  organic  organization  or  by  a contractor.  The  following 
are  primary  QA  functions: 

a.  Planning. 

b.  Development  of  QA  Policies  and  Procedures. 

c.  Configuration  Audits. 

5-4.  PLANNING.  QA  planning  starts  with  a review  of  the 
DAR,  DPD,  DPP,  and  FD.  This  review  ends  in  a QA  plan  which 
describes  the  required  quality  assurance  functions,  responsi- 
bilities, and  tools  to  make  sure  that  software  is  reliable 
and  can  be  maintained.  The  QA  plan  is  coordinated  with  the 
project  manager. 

5-5.  POLICIES,  PROCEDURES,  AND  STANDARDS. 

a.  Performance  of  QA  activities  requires  establishment 
of  QA  policies  and  procedures.  QA  policies  and  procedures 
outline  the  rules  for  conducting  configuration  audits  and 
assuring  compliance  with  ADP  standards  requirements  and  local 
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procedures  for  design,  development,  and  test  of  computer 
programs,  and  software  deliveries. 

b.  QA  personnel  assist  in  the  adaptation  and  interpreta- 
tion of  directed  and  local  ADP  standards.  Relaxation  from 
standards  may  be  required  due  to  technical  problems  and  j 

schedule  impacts.  Deviations  from  standards  must  be  coordi-  ; 

I 

nated  with  QA  personnel.  \ 

5-6.  CONFIGURATION  AUDITS.  To  make  sure  QA  policies  and  pro- 

i 

I 

cedures,  and  ADP  standards  are  followed,  QA  personnel  conduct 

the  Functional  Configuration  Audits  (FCA)  and  the  Physical  j 

I 

Configuration  Audits  (PCA)  as  described  in  Chapter  4 . QA  '■ 

\ 

personnel  will  document  findings  and  recommendations.  ! 

5-7.  QA  PARTICIPATION  IN  REVIEWS.  QA  personnel  should  ' 

participate  in  the  SRR,  SDR,  PDR,  CDR,  and  PVR  to  insure  the 
following: 

a.  Documentation  complies  with  directed  and  local 
standards. 

b.  Assist  other  project  participants  in  determining 
technical  adequacy  of  documentation. 

c.  Insure  project  schedules  include  adequate  time  for 
the  conduct  of  configuration  audits. 

5-8.  TEST  PHASE  SUPPORT.  Specific  QA  tasks  to  be  performed 
during  the  test  phase  are  as  follows: 


a.  Make  sure  master  copies  of  test  procedures,  test 
results,  and  test  reports  are  maintained. 

't: 

< b.  Review  test  results  for  consistency  with  test  criteria. 

c.  Monitor  all  tests  to  insure  actual  tests  performed 
are  as  specified  in  documented  test  procedures. 

d.  Compare  the  configurations  of  hardware/software 
components  utilized  in  the  test  against  the  configuration 

T identified  in  the  test  procedures. 

• e.  Certify  that  the  test  output  is  correct,  the  test 

satisfies  requirements,  and  the  test  data  package  is 
complete. 

f.  Retain  certified  test  data  packages  containing  a 
copy  of  test  procedures,  test  output,  and  discrepancy  reports. 


g.  Insure  compliance  with  approved  quality,  test  and 
configuration  management  procedures. 


Chapter  6 
TEST  MANAGEMENT 

6-1.  GENERAL.  Methods  and  procedures  for  testing  ADSs  are 
described  in  this  chapter.  Details  of  the  testing  approach 
include  test  validation  methods,  test  control,  and  test 
reporting.  In  addition,  testing  aspects  that  affect 
the  user  and  other  organizations  are  identified.  While  the 
details  provided  are  generally  geared  to  large  ADS  develop- 
ment projects,  a subset  of  these  procedures  can  be  tailored 
to  the  smallest  of  development  efforts. 

6-2.  PURPOSE.  To  discuss  guidelines  for  establishing  and 
implementing  the  test  program. 

6-3.  INFORMATION.  The  test  program  has  two  major  phases; 
development  testing  and  validation  testing.  Development 
testing  includes  module  and  integration  testing.  Validation 
testing  is  conducted  by  a test  group  independent  of  the 
developer  during  the  test  phase.  A variation  of  validation 
testing  has  been  successfully  implemented  on  large  multisite 
ADS  development  projects.  This  variation  is  described  in 
paragraphs  6-6  and  6-7,  and  is  composed  of  two  parts. 

Part  One  is  an  Environmental  System  Test,  Phase  I (EST-I) , 
conducted  in  as  near  a "real  world"  environment  as 
possible  Part  Two  is  an  Environmental  System  Test,  Phase 
II  (EST-II)  conducted  at  one  or  more  "live"  operational 
sites  by  the  users.  Both  EST-I  and  EST-II  may  be  used  to 
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provide  validation  testing  functions  and  lead  to  an  opera- 
tional capability. 

*'  6-4.  DEVELOPMENT  TESTING. 

a.  GENERAL.  Development  testing  is  conducted  by  the 
programmers  and  analysts  responsible  for  development  of  the 
CPCI.  Testing  is  initiated  when  a particular  module  has 
been  coded,  compiled,  and  verified  to  the  extent  that  the 
module  accepts  normal  inputs  and  provides  outputs.  During 
development  testing,  emphasis  is  placed  on  testing  discrete 
logic  sequences;  determining  accuracy  of  computation;  showing 
repeatability  of  result;  testing  upper,  lower,  and  nominal 
ranges  of  data  values;  testing  error  conditions;  veri- 
fying timing,  sizing,  interfaces,  and  data  handling  charac- 
teristics on  a module-by-module  basis;  and  integrating 
the  modules  and  test  the  CPCI  as  a whole. 

b.  DEVELOPMENT  TEST  PLAN.  All  development  test  activities 
are  conducted  against  development  test  plans.  These  plans  are 
prepared  by  the  development  programmers  and  analysts  during 
the  development  phase  and  form  the  basis  for  the  test  plan  (PT) , 
described  in  DOOM  4121. 17M.  Normally,  development  test  plans 
are  prepared  prior  to  the  prelimi.nary  design  review  (PDR)  and 
test  procedures  added  prior  to  the  critical  design  review 
(CDR) . They  should  be  reviewed  by  appropriate  project  members 
prior  to  the  start  of  development  testing. 

(1)  Development  test  plans  (Attachment  2)  are 
issued  for  each  CPCI,  in  accordance  with  project  schedules. 
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(2)  The  intent  is  to  have  a well  coordinated 
test  program  to  insure  that  complete  and  verified  software 
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V is  delivered  to  the  independent  test  group  at  the  completion 

T 

^ of  development  testing.  In  many  cases,  the  highest  level  of 

development  tests  are  dry  runs  for  the  validation  tests  that 
follow.  Development  test  plans  should  be  coordinated  within 
the  development  organization  to  insure  these  goals  are  achieved.  j 

c.  DEVELOPMENT  TEST  REPORTING.  The  project  manager  j 

should  be  provided  a report  on:  the  number  of  tests  executed, 

‘ analyzed,  and  validated;  tests  with  problems;  tests  to 

be  rerun;  assessment  of  problems  on  the  schedule;  etc.  This 
reporting  process  is  to  gain  insight  as  to  the  progress  of 

I 
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testing  amd  to  determine  if  the  schedule  is  being  met.  Any 
change  made  to  the  software  or  to  its  documentation  as  a result 
of  testing  is  noted  and  incorporated  in  an  update  of  the 
program  specification  (PS)  to  be  issued  after  development 
testing. 

6-5.  VALIDATION  TESTING.  This  is  the  formal  testing  that 
occurs  prior  to  delivery  of  the  software.  It  is  normally 
performed  by  a test  group,  independent  of  the  developers. 

Emphasis  is  on  validating  that  the  requirements  of  the  FD  and 
the  SS  are  met.  The  main  purpose  of  this  testing  is  to  demon- 
strate that  actual  performance  meets  required  performance  in  as 


nearly  an  operational  environment  as  is  practical.  All 
validation  testing  should  be  conducted  using  operational 
system  software  and  hardware.  Sufficient  resources  and 
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time  must  be  allocated  to  the  independent  test  group. 

a.  TEST  PLAN  (PT) . The  PT  defines  the  test  program 

to  be  conducted  by  the  independent  test  group.  It  specifies 
the  purpose,  objectives,  and  overall  description  of  testing 
to  be  performed.  Detailed  test  procedures  are  also  included 
which  govern  the  conduct  of  individual  tests.  The  PT  is 
prepared  by  the  development  activity  and  coordinated  with  the 
independent  test  group  prior  to  initiation  of  the  test  phase. 
The  independent  test  team  may  augment  the  PT  with  additional 
tests,  as  required.  The  PT  is  prepared  in  accordance  with 
DOOM  4120. 17M. 

b.  TEST  EXECUTION. 

(1)  Test  Teams.  The  independent  test  group  usually 
is  organized  into  test  teams,  with  each  team  being  given  the 
responsibility  for  validating  a selected  subset  of  the  ADS. 

To  insure  the  test  teams  become  as  familiar  as  possible  with 
their  subset,  the  assignment  of  test  teams  should  be  made 
early  enough  to  allow  each  team  to  become  thoroughly  knowl- 
edgeable in  the  functions  and  operation  of  its  subset  of  the 
software.  Additional  personnel  should  be  added  to  the  teams 
as  the  project  matures.  Each  test  team  consists  of  members 
of  the  independent  test  group,  with  one  designated  as  team 
leader.  Generally,  it  is  beneficial  to  augment  the  test 
teams  with  several  developers  and  users  who  can  support  the 
analysis  of  test  runs.  However,  responsibility  for  testing 
rests  with  the  test  team  leader. 


5*6 


(2)  Control  of  Test  Execution.  ) 

i 

i 
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(a)  The  test  plan  contains  a master  schedule  j 

i 

of  the  planned  tests.  This  schedule  indicates  the  earliest  | 

planned  execution  date,  and  provides  for  a reasonable  number  ] 

I 

of  reruns  of  each  case.  The  schedule  is  prepared  on  the 
basis  of  critical  test  dependencies,  computer  time  require- 
ments and  availability. 

(b)  During  testing,  a record  should  be  main- 
tained of  the  number  of  times  a test  is  executed  and  the 
reasons  for  reruns.  The  software  configuration  used  for 
each  run  should  be  identified.  Status  of  the  analysis 

of  each  test  is  also  recorded  to  give  the  test  group,  the 
development  group,  and  the  project  manager  visibility 
into  the  progress  of  testing. 

(c)  For  large  development  projects,  it  is 
advantageous  to  establish  a test  review  board  (TRB)  composed 
of  developer  and  user  representatives.  This  board  performs 
the  primary  test  monitoring  function  during  formal  testing, 
and  expedites  any  corrective  action  required. 

(d)  Changes  in  the  data  base  which  are  required 
to  facilitate  testing  must  be  coordinated  through  the  software 
developers,  the  project  manager  and,  in  some  instances,  the 
user.  The  data  test  package  should  contain  a record  of 
the  data  base  configuration  used  for  each  test. 
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(e)  The  test  procedures  are  the  key  documents 
to  be  controlled  during  the  test  phase.  Approval  authority 
for  changes  to  test  procedures  depends  on  the  extent  of  the 
change.  Any  change  that  relates  to  specification  require- 
ments must  be  approved  by  the  configuration  control  board 
(CCB) , while  the  test  review  board  or  test  team  leader  may  be 
the  approval  authority  for  all  other  changes  that  do  not 
have  a cost  and  schedule  impact, 
e.  TEST  REPORTING. 

(1)  A test  status  report  should  be  prepared  and 
distributed  as  frequently  as  the  size  of  the  project  requires. 

The  material  in  this  report  will  be  essentially  the  same  as 
that  presented  in  test  review  board  (TRB)  meetings,  if  the 
project  has  established  such  a board.  For  each  reporting 
period,  the  test  status  report  will  identify: 

(a)  Tests  executed. 

(b)  Tests  analyzed  and  validated. 

(c)  Tests  having  open  software  problems. 

(d)  Tests  to  be  rerun. 

(e)  Status  of  test  reruns. 

(f)  Peview  of  open  software  problems. 

(g)  Assessment  of  impact  of  open  software 

problems. 

(h)  ■ Forecast  of  tests  to  be  executed  during 
next  reporting  period. 

(i)  Summary  of  complete  test  schedule. 

a, 


(2)  A supplemental  report  which  is  essential  during 
testing  is  the  software  problem  summary.  Each  Software 
Problem  Report  (SPR)  written  during  testing  is  identified  by 
its  control  number,  the  module  affected,  the  date  the  SPR 
was  logged,  and  the  date  the  SPR  was  closed. 

(3)  The  final  reporting  instrument  for  each  test 
procedure  is  Section  2 of  the  test  analysis  report  (RT)  . 

When  analysis  has  been  completed,  the  final  test  procedure, 
test  results  and  any  change  control  documentation  generated 
as  a result  of  the  test  execution,  will  be  provided  to  those 
personnel  responsible  for  preparation  of  the  final  RT.  At  the 
conclusion  of  all  testing,  the  RT  is  finalized  and  used  as  the 
primary  input  to  the  SVR. 

6-6.  ENVIRONMENTAL  SYSTEM  TEST,  PHASE  I (EST-I) . 

a.  PURPOSE.  The  primary  purpose  of  the  EST-I  is  to 
insure  the  CPCI  meets  its  specified  requirements  in  a simulated 
"live"  environment  prior  to  operational  release  or  further 
testing.  EST  I also  allows  an  organization  independent  of 

the  developer  to  perform  the  testing,  check  systems  integra- 
tion, and  measure  the  impact  on  Data  Processing  Installation 
(DPI)  operations.  Successful  completion  of  EST-I  results 
in  update  of  the  product  baseline  and  certification  to 
release  to  EST  II  or  for  operational  use. 

b.  PERSONI:eL  composition/participation.  The  participants 
in  EST  I are  quality  assurance  personnel,  development 
activity  representatives,  and  user  representatives. 


C . INPUTS/MATERIALS . 

(1 ) Test  Plan  (PT) . 

(2)  Users  Manual  (UM) . 

(3)  Program  Maintenance  Manual  (MM) . 

(4)  Computer  Operation  Manual  (OM) . 

(5)  Computer  programs  and  test  data. 

(6)  Other  documents/data,  as  required  locally. 

d.  ACTIONS/DECISIONS. 

(1)  Examine  documentation  for  compliance  with 
standards. 

(2)  Perform  tests  in  accordance  with  the  PT. 

(3)  Analyze  test  results. 

(4)  Prepare  a Test  Analysis  Report  (RT) . 

(5)  Assist  with  the  retest  of  corrected  errors. 

e . OUTPUTS/MATERIALS . 

(1)  A Test  Analysis  Report  (RT) , in  accordance  with 
DOOM  4120. 17M. 

(2)  System  for  release  to  EST-II  or  for  operational 

use . 

(3)  Updated  Product  Baseline. 

(4)  Other  documents/decisions  as  required  locally. 

f.  CONDUCT  OF  EST-I.  The  QA  activity  is  responsible  for 
the  conduct  of  EST-I.  The  EST-I  Test  Analysis  Report  iden- 
tifies problem  areas  which  must  be  corrected  prior  to  retest- 
ing. The  developer  and  the  user  have  the  opportunity  to 
review  all  test  results  to  assure  the  products  satisfactorily 
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meet  the  functional  requirements.  Corrections,  if  necessary,  i 

are  made  to  the  ADS  programs  and  documentation,  and  the 
system  undergoes  additional  testing,  as  required.  When 
tests  meet  with  the  approval  of  all  participants,  the  pro- 
duct baseline  is  updated. 

6-7.  ENVIRONMENTAL  SYSTEM  TEST,  PHASE  II  (EST-II) . 

a.  PURPOSE.  The  purpose  of  EST-II  testing  is  to  per- 
form an  evaluation  of  computer  programs,  documentation,  and 
output  products  in  a "live"  environment  at  one  or  more 
selected  operational  sites.  It  allows  base/MAJCOM  data 
automation  and  functional  personnel  to  perform  their  indepen- 
dent evaluations  of  the  system  with  minimum  assistance  from 
the  development  activity. 

b.  PERSONNEL  COMPOSITION/PARTICIPATION.  The  participants 
are  quality  assurance  personnel,  developer  personnel,  site 
personnel,  user  personnel,  and  when  required,  MAJCOM  repre- 
sentatives. 

C . INPUTS/MATERIALS . 

(1)  Test  Plan  (PT) . 

(2)  Users  Manual  (UM) . 

(3)  Progrcun  Maintenance  Manual  (MM)  . 

(4)  Computer  Operation  Manual  (OM) . 

(5)  System/Subsystem  Release  (Object  Prograuns)  . 

(6)  Live-Test  Data. 

(7)  Other  documents/data  as  required  locally. 


d.  ACTIONS/DECISIONS.  The  developer,  in  conjunction 


vilth  QA  personnel  and  the  users,  nominates  test  sites. 

The  test  is  conducted  under  the  auspices  of  the  QA  activity 
with  assistance  of  the  developer,  the  base  level  functional 
and  data  automation  personnel,  and,  when  required,  MAJCOM 
functional  and  data  automation  personnel.  Problems  are 
documented  and  corrections  are  validated  by  QA  prior  to 
release.  When  a successful  System  Validation  Review  (SVR) 
has  been  completed,  the  ADS  is  released. 

e.  OUTPUTS/MATERIALS. 

(1)  A Test  Analysis  Report  (RT) . 

(2)  Recommendation  for  conducting  the  SVR  or  further 

testing. 

(3)  Other  documents/decisions  as  required  locally. 

f.  CONDUCT  OF  EST-II.  The  developer,  in  conjunction 
with  the  QA  activity  and  the  user,  nominates  test  sites. 
Coordination  with  MAJCOM  and  base  level  functional  and 
data  automation  personnel  is  required  to  identify  site 
adequacy  to  support  the  test  requirements.  The  testing 
to  be  performed  provides  an  evaluation  of  the  capability 
of  the  ADS  to  operate  in  a "real  world"  environment  and 
establishes  the  adequacy  of  the  ADS  to  the  user.  The  MAJCOM/ 
base  level  data  automation  and  functional  personnel  will  perform 
an  independent  evaluation  of  the  ADS  with  minimvim  assistance 
from  the  developer.  Problems  will  be  documented  and  correc- 
tions validated  by  QA  prior  to  release.  When  the  test 


activity  is  completed,  the  SVR  will  be  conducted.  Approval 
by  the  development  activity  and  the  user  representative  is 
required  to  successfully  complete  the  SVR.  Following  the 
SVR,  the  ADS  is  released  for  operational  use. 
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PART  I 

ADS 

BCR 

CCB 

CDR 

Cl 

CM 

CMP 

CPC  I 

DAR 

DBCR 

DPD 

DPI 

DPP 

DPR 

DS 

DT 

EST  I 
EST  II 
FCA 
FD 


- ABBREVIATIONS 

Automated  Data  System 
Baseline  Change  Request 
Configuration  Control  Board 
Critical  Design  Review 
Configuration  Item 
Configuration  Management 
Configuration  Management  Plan 
Computer  Program  Configuration  Item 
Data  Automation  Requirement 
Data  Base  Change  Request 
Data  Project  Directive 
Data  Processing  Installation 
Data  Project  Plan 
Design  Problem  Report 
Data  Base  Specification 
Development  Test  Plan 
Environmental  System  Test,  Phase  I 
Environmental  System  Test,  Phase  II 
Functional  Configuration  Audit 
Functional  Description 
Full  Operational  Capability 


FOC 


HIPO 


MM 

OM 

OS&M 

PCA 

PDR 

PS 

PT 

PVR 

QA 

RD 

RT 

SDR 

SPR 

SRR 

SS 

SVR 

TRB 

UM 
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Hierarchy  Input  Process  Output 
Program  Maintenance  Manual 

Computer  Operation  Manual  ] 

Operational  Support  and  Maintenance 

Physical  Configuration  Audit 

Preliminary  Design  Review 

Program  Specification 

Test  Plan 

Product  Verification  Review 

Quality  Assurance 

Data  Requirements  Document 

Test  Analysis  Report  ' 

System  Design  Review 
Software  Problem  Report 

System  Requirements  Review  i 

i 

System/Subsystem  Specifications  | 

I 

System  Validation  Review  i 

i 

Test  Review  Board  ] 

Users  Manual  j 

I 

i 
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PART  II  - DEFINITIONS 


allocated  baseline  - The  initial  approved  allocated  con- 
figuration identification  established  at  end  of  the 
definition  phase. 

baseline  - A configuration  identification  document  or  set  of 
such  documents  formally  designated  and  fixed  at  a specific 
time  during  a CPCI's  life  cycle.  Baselines,  plus  approved 
changes  to  those  baselines  constitute  the  current  configura- 
tion identification. 

baseline  change  request  (BCR)  - A request  for  alteration  in 
the  configuration  of  a computer  program  configuration  item 
that  is  delivered  or  under  development,  after  formal 
establishment  of  its  configuration  identification. 
computer  program  configuration  item  (CPCI)  - An  ADS  or  portion 
of  an  ADS  that  is  designated  for  configuration  management. 
configuration  audit  - a process  to  verify  conformance  to 
specifications  and  standards. 

configuration  control  - The  systematic  evaluation,  coordination, 
approval  or  disapproval,  and  implementation  of  approved  changes 
in  the  configuration  of  a CPCI  after  formal  establishment  of  its 
configuration  identification. 

configuration  control  board  (CCB)  - A board  composed  of 
representatives  from  program/project  office  and  using/sup- 
porting organizations. 

configuration  identification  - The  currently  approved  tech- 
nical description  of  the  CPCI  or  ADS. 
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configuration  item  (Cl)  - An  item  of  ADPE  that  is  designated 
for  configuration  management. 

configuration  management  (CM)  - A management  discipline  that 
applies  technical  and  administrative  direction  and  surveil- 
lance to: 

(a)  identify  and  document  the  functional  and  physical 
characteristics  of  a configuration  item. 

(b)  control  changes  to  those  characteristics. 

(c)  record  and  report  configuration  status. 
configuration  management  plan  (CMP)  - A document  which 
describes  project  responsibilities  and  procedures  for  imple- 
menting CM. 

configuration  status  accounting  - The  recording  and  reporting  of 
the  approved  configuration  identification,  the  status  of  the 
proposed  changes  to  the  approved  configuration,  and  the 
implementation  status  of  approved  changes. 
critical  design  review  (CDR)  - A formal  review  conducted 
during  the  development  phase  before  translating  logic,  and 
algorithms  to  coded  instructions . 

data  base  change  request  (DBCR)  - A form  used  to  initiate 

and  control  data  base  changes  after  the  data  base  is  placed 
under  configuration  control. 
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design  problem  report  (DPR)  - A form  used  for  documenting 
problems  identified  during  reviews  and  audits. 
development  testing  - Testing  of  computer  programs  by  the 
development  programmers  and  analysts  prior  to  EST  I. 
development  test  plan  (DT)  - A document  which  specifies  the 
method  and  content  for  development  testing  from  the  lowest 
compilable  level  up  through  the  complete  computer  program 
configuration  item.  Defines  test  management,  reports, 
controls,  manpower,  acceptance  criteria,  and  test  procedures. 
deviation  A written  authorization,  granted  prior  to 
the  development  of  a CPCI,  to  depart  from  a particular 
performance  or  design  requirement;  a specification  for  a 
specific  number  of  units;  a specific  period  of  time;  or 
established  standards. 

functional  baseline  - The  initial  approved  functional  con- 
figuration identification. 

functional  configuration  audit  (FCA)  - The  formal  examination 
of  CPCI  to  verify  that  the  performance  specified  in  the 
SS  has  been  achieved, 

module  - A program  unit  that  is  discrete  and  identifiable 
with  respect  to  compiling  and  combining  with  other  units. 
physical  configuration  audit  (PCA)  - The  formal  examination  of 
the  coded  version  of  a computer  program  configuration  item 
against  its  technical  documentation. 


preliminary  design  review  (PDR)  A formal  review  of  the 
subsystem  design  approach  for  a CPCI  occuring  between  the 
SDR  and  CDR. 


product  baseline  - The  initial  approved  product  configuration 
identification. 

product  verification  review  (PVR)  - A formal  review  conducted 
by  the  developer  for  each  CPCI  at  the  end  of  the  development 
phase  to  establish  the  Product  Baseline  for  that  CPCI  and  to 
insure  preparation  for  the  Test  Phase  has  been  completed. 
quality  assurance  (QA)  - All  actions  that  are  taken  to 
assure  that  a development  organization  delivers  products 
that  meet  performance  requirements  and  adhere  to  standards 


and  procedures. 

software  problem  report  (SPR)  - A form  used  to  report  a 
suspected  or  existing  discrepancy  or  deficiency  in  an  exist- 
ing computer  program,  its  operational  documentation  or 
interfacing  hardware. 

specification  - A document  that  describes  the  requirements 
for  the  development  or  acquisition  of  ADPE  and/or  software. 
system  design  review  (SDR)  - A formal  review  of  the  system 
design  approach  for  an  ADS. 

system  requirements  review  (SRR)  - A formal  review  of  the 
requirements  for  an  ADS. 

system  validation  review  (SVR)  - A formal  review  of  the  results 
of  the  Test  Phase  to  insure  that  the  ADS  satisfies  the  require- 
ments of  the  SS  and  FD. 
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test  analysis  report  (RT)  - A document  containing  the  results 
and  analyses  of  tests  executed  during  the  Test  Phase. 
waiver  - A written  authorization  to  accept  a configuration 
item  or  other  designated  item  that  has  been  found  to  depart 
from  specified  requirements,  but  nevertheless  is  considered 
suitable  for  use  as  is  or  after  rework  by  an  approved  method. 
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DEVELOPr'^NT  TEST  PLAN 

The  purpose  of  the  development  test  plan  is  to  describe  the 
testing  which  is  to  be  done  during  integration  testing. 
Chapter  1 - GENERAL  INFOR^-IATION 

1-1.  Summary.  Summarize  the  purpose,  scope,  and  intended 
use  of  the  development  test  plan. 

1- 2.  References.  List  applicable  project  documentation. 
Chapter  2 - TEST  MANAGEMENT 

2- 1 . Roles  and  Responsibilities.  Describe  the  roles  and 
responsibilities  of  test  participants  in  the  development 
organization  during  the  testing  phase. 

2- 2.  Reporting  Procedures.  Describe  test  reporting  proce- 
dures to  be  employed. 

Chapter  3 - TEST  CONTROL. 

3- 1.  Procedures.  Establish  test  control  procedures. 
Address; 


a.  The  control  and  accounting  system  for  test  proce- 
dures, test  data,  and  test  execution, 

b.  Procedures  for  increasing  control  over  software 
products  as  integration  testing  progresses. 

c.  Procedures  for  retest. 


Chapter  4 - TEST  DEFINITION 

This  section  defines  each  test,  acceptance  criteria  for 

the  test,  and  procedures  for  conducting  the  test.  The  format 
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for  presentation  of  test  definitions  is  as  follows: 

4-1.  Test  Description.  Describe  the  purpose  of  the  test, 
required  inputs/outputs,  and  acceptance  criteria  for  mea- 
suring success  of  the  test. 

4-2.  Test  Identification.  Assign  a unique  identifier 
and  title  for  each  test. 

4-3.  Environment.  Describe  the  hardv/are  and  other  soft- 
ware required  to  execute  the  test. 

4-4.  Test  Procedures.  Describe  the  procedures  for  con- 
ducting the  test. 

Chapter  5 - RESOURCE  REQUIREMENTS 

Discuss  the  equipment,  personnel,  and  facilities  that  will 
be  required  to  perform  integration  testing. 

Chapter  6 - SCHEDULES 

Provide  a detailed  schedule  for  development  testing.  The 
schedule  should  identify  when  individual  modules  are  inte- 
grated into  the  CPCI. 


AFR  300-15 


Attachment  3 

i ' 

if- 

1 

PROGRAM  MAINTENANCE  MANUAL 

PURPOSE;  The  primary  purpose  of  program  maintenance  manual 
is  to  facilitate  transfer  of  programs  between  individuals  at 
any  point  in  the  life  cycle  of  the  programs.  A secondary 
purpose  is  to  rapidly  determine  program  status  at  any  point 
during  development.  The  following  guidelines  are  provided: 

a.  A program  maintenance  manual  (MM)  should  be  kept  for 
every  module. 

b.  The  MM  is  prepared  and  maintained  by  the  programmer 
or  programmer  team. 

c.  Specific  format  for  the  MM  should  be  provided  by 
the  development  organization  using  DOD  Manual  4120. 17M  as  a 
guide. 

d.  General  guidelines  for  content  include; 

(1)  Requirements  Section.  This  section  should 
include  extracts  from  the  FD  and  SS  pertinent,  to  the  module. 
Additionally,  a narrative  should  be  included  which  explains 
how  the  module  satisfies  the  requirements. 

(2)  Constraints  Section.  This  section  should 
include  all  known  constraints,  e.g.,  required  core  storage, 
peripherals  required,  table  lengths,  etc. 


(3)  Interface  Section.  Both  internal  and  external 
interfaces  should  be  included  in  this  section.  Extracts  from 
the  RD  and  DS  should  be  used,  if  available. 

(4)  Design  Section.  Include  logic  or  structure  charts 
which  show  the  complete  logic  flow  of  the  module  as  well  as 
all  interface  elements.  The  PS  should  be  prepared  from  these 
charts  if  required  by  project  management.  A current  program 
listing  should  also  be  included  in  this  section. 

(5)  Test  Section.  This  section  includes  test  instructions 
and  a description  and  the  location  of  all  items  necessary  for 
the  conduct  of  module  testing. 
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CONFIGURATION  MANAGEMENT  PLAN  FORMAT 

Section  I - INTRODUCTION. 

State  the  purpose  of  the  configuration  management  plan  (CMP) 
and  briefly  describe  its  contents. 

Section  II  - PROJECT  ORGANIZATION. 

Describe  the  project  organization  and  its  relation  to  com- 
mand structure.  Identify  the  organizational  level  at  which 
the  configuration  management  function  will  be  performed.  A 
checklist  for  the  preparation  of  this  section  is  as  follows: 

a.  Is  configuration  management  at  a suitable  level  in 
the  project  structure? 

b.  Are  responsibilities  clearly  defined? 

c.  Are  sufficient  resources  identified  to  perform  the 
required  tasks? 
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d.  Is  there  adequate  configuration  management  represen- 
tation on  the  CCB? 

Section  III  - CONFIGURATION  IDENTIFICATION. 

Identify  responsibilities  for  establishing  configuration 
identification.  Identify  the  CPCI (s)  to  be  controlled  and 
related  documentation.  Identify  responsibilities  for  pre- 
paring the  documentation  plan.  A checklist  for  the  prep- 
aration of  this  section  is  as  follows: 

a.  Does  the  CMP  identify  the  documents  that  establish  the 
baselines? 

b.  Are  the  responsibilities  for  preparing  and  approving 
project  documentation  clearly  defined? 

c.  Does  the  documentation  plan  indicate  when  project 
documentation  is  placed  under  configuration  control? 

d.  Does  the  schedule  in  the  documentation  plan  meet 
project  and  user  requirements? 

e.  Does  the  CMP  include  an  appropriate  numbering  scheme 
for  the  software  products  produced? 
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Section  IV  - CONFIGURATION  CONTROL. 


Define  procedures  for  control  of  CPCIs  and  for  processing 
changes  thereto.  Include  change  control  procedures  for  proces- 
sing changes  generated  by  the  development  activity  and  other 
organizations.  A checklist  for  the  preparation  of  this  section 
is  as  follows: 


a.  Does  the  CMP  include  an  adequate  change  control  system 
for  the  various  classes  of  changes  established? 


b.  Is  a method  provided  for  controlling  the  interfaces 
between  elements  of  the  CPCI  during  development  and  testing? 

c.  Does  the  CMP  identify  methods  for  controlling  inter- 
faces with  external  ADS? 

d.  Is  the  change  control  process  clearly  described  with 
flow  charts  and  sample  forms? 

s.  Is  there  an  adequate  procedure  described  for  storage 

and  physical  control  of  software  products? 

f.  Is  there  an  adequate  procedure  described  for  release 
of  software  products  into  a controlled  environment? 
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• Section  V - CONFIGURATION  STATUS  ACCOUNTING. 

Describe  the  recording  and  reporting  procedures  and  forms  to 
be  used  for  configuration  status  accounting.  Define  any  auto- 
mated methods  to  be  used  in  status  reporting.  A checklist  for 
the  preparation  of  this  section  is  as  follows: 

a.  Does  the  CMP  identify  the  data  required  for  configura- 
tion status  accounting,  and  the  method  for  recording  it? 

b.  Does  it  identify  the  status  accounting  reports  j 

required,  and  their  frequency  and  distribution?  > 

c.  Are  forms,  formats,  and  reports  clearly  identified 
and  included? 

Section  VI  - REVIEWS  AND  AUDITS. 

Describe  plans  for  configuration  management  support  to  reviews 
and  audits.  Define  products  and  documents  to  be  reviewed  or 

audited,  reviewing  authority,  method  for  handling  deviations 
or  waivers,  change  procedures  and  forms,  and  numbering  changes. 

Section  VII  - OPERATIONAL  IMPLEMENTATION. 

Define  procedures  for  maintaining  the  product  baseline  after 
initiation  of  the  operation  phase. 


AFR  300-15 


Attachment  5 


TYPES  OF  CONFIGURATION  STATUS 
ACCOUNTING  LOGS 


1.  PRODUCT  LOG;  The  product  log  is  used  to  list  and 
describe  existing  tapes,  decks,  and  listings  of  released 
software: 

a.  Tape  identification. 

b.  Content  of  tape,  e.g.,  module  IDs,  CPCI  identifi- 
cation, etc. 

c.  Listings  and  card  decks  controlled  and  numbered  by 
the  module  ID. 

d.  Originator  name  and  date.  Tape  library  number  (if 
applicable) . 

2.  MODULE  LOG.  The  module  log  is  a history  of  each  module's 
development. 

a.  Module  ID. 

b.  List  the  BCR/SPR/DBCR  that  resulted  in  changes  to 

the  module. 

c.  Tape  number  and  tape  library  number,  if  applicable. 

d.  Prograun  specification  (PS),  title  and  number,  in 
which  the  module  is  described. 

3.  SOFTWARE  PROBLEM  REPORT  (SPR)  LOG.  The  SPR  log  is  used 
to  list  and  describe  all  SPR  data,  including  fixes  and  other 
closing  information; 
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a.  SPR  number  and  date. 
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b. 

c. 

d. 

e. 


SPR  originator ' s naune . 

Module  ID  and  and  data  base  information. 
Organization  responsible  for  action. 
Status. 


DATA  BASE  CHANGE  REQUEST  (DBCR)  LOG.  The  DBCR  log  is 


used  to  list  and  describe  all  DBCRs: 

a.  DBCR  number  and  date. 

b.  DBCR  originator's  name. 

c.  Data  base  identification. 

d.  Identification  and  date  of  data  base  that  incorporates 
the  change. 

5.  BASELINE  CHANGE  REQUEST  (BCR)  LOG.  The  BCR  log  is  used 
to  list  BCRs. 

a.  BCR  number  and  date. 

b.  Originator's  naune. 

c.  BCR  classification  (Class  I or  II)  . 

d.  Baseline  on  which  BCR  was  originated. 

e.  Baseline  documentation  affected. 

f.  Approval/disapproval  and  date. 

g.  Date  implemented,  if  approved. 


I 


6.  DESIGN  PROBLEM  REPORT  (DPR)  LOG.  The  DPR  log  is  used  to 
list  all  DPRs  submitted  during  reviews  and  audits: 
a.  DPR  number  and  date. 

[ 

j b.  Originator. 

1 

I c.  Organization  responsible  for  action. 


^ ^ 

AFR  300-15 
Attachment  6 

CONFIGURATION  CONTROL  FORMS 
PURPOSE:  To  provide  instructions  for  completion  of 

configuration  control  forms. 

a.  DESIGN  PROBLEM  REPORT  (DPR) 

CONTROL  NUMBER;  Enter  a unique  control  number  in 
accordance  with  local  configuration  management  procedures. 

DATE:  Enter  date  submitted. 

ORIGINATOR;  Individual  who  originated  the  DPR,  his 
organization,  and  phone  number. 

DOCUMENT;  Identify  the  document  in  which  the 
problem  exists. 

PAGE/SECTION/FIGURE;  Identify  the  specific 
reference  in  the  document  cited  above. 

PROBLEM/RECOMMENDATION;  Describe  the  problem  and 
recommendation  resulting  from  the  review  in  sufficient 
detail  for  corrective  action  to  be  taken. 

ACTION  CATEGORY;  Reviewing  development  organization 
check  one  of  four  indicated  categories. 

ACTION  ITEM  NUMBER;  Number  assigned  in  accordance 
with  local  procedures  as  a cross  reference  to  the  minutes  of 
the  review. 

DISPOSITION;  Indicate  action  taken  or  to  be  taken 
to  resolve  the  DPR. 


DESIGN  PROBLEM  REPORT  (DPR) 
Originator;  


Document  

Page/Section/Figure  


Problem/Recommendation 


CONTROL  NO. 
DATE 


Action  Category 

Design 

1 . 

n 

Incorrect 

2 . 

□ 

Incomplete 

3. 

□ 

Action  Item  No. 

Editorial 

4. 

0 

Disposition: 

Closed  [] 

b.  SOFTWARE  PROBLEM  REPORT  (SPR) 

CONTROL  NUMBER:  Enter  a unique  control  number  in 
accordance  with  local  configuration  management  procedures. 

DATE:  Enter  date  SPR  submitted. 

TO:  Enter  the  organization  responsible  for  development 
or  maintenance  of  the  software. 

FROM:  Enter  the  initiating  organization. 

INFO  COPIES  TO:  Self-explanatory. 

PROGRAM  NAME:  Enter  the  name  of  the  program  in 
which  the  problem  or  discrepancy  was  detected. 

IDENT:  Enter  the  identification  of  the  program 

involved . 

ADS:  Enter  the  title  of  the  ADS  of  which  the 

program  is  a part. 

RUN  DATE:  Enter  the  date  the  program  was  run  in 
which  the  discrepancy  or  error  was  detected. 

POINT  OF  CONTACT:  Enter  the  name  of  the  individual 
in  the  organization  which  initiated  the  SPR  who  is  most 
familiar  with  the  problem, 

PROBLEM  DESCRIPTION:  Describe  what  the  discrepancy 
or  error  is  and  circumstances  which  contributed  -i  o the 
cause.  Describe  what  should  have  occurred  had  there  been  no 
discrepancy  and  the  impact  of  the  discrepancy. 

COMMENTS:  Indicate  the  urgency  of  the  correction 


and  any  other  pertinent  facts. 


PROBLEM  ANALYSIS:  Describe  the  cause  of  the 


discrepancy  or  error/  the  impact/  and  any  other  programs  or 
data  bases  affected. 

RECOMMENDED  ACTION:  Describe  the  proposed  corrective 
action  (if  necessary) , and  provide  an  estimate  of  the  time 
and  resources  required  to  complete  the  recommended  action. 

APPROVED:  Self-explanatory. 

DISAPPROVED : Sel f -explanatory . 

SIGNATURE:  Self-explanatory. 

DATE;  Enter  date  of  signature. 

ACTION  TAKEN;  Indicate  what  corrective  action  was 
taken  if  such  action  was  necessary.  Additionally/  include 
the  date  the  action  was  completed. 


Software  Problem  Report 


Problem  Analysis  Analyst 


Date 


Recommended  Action 


I 


Q Approved 

Q Disapproved 

Signature 

Date 

Action  Taken 


C.  DATA  BASE  CHANGE  REQUEST  (DBCR) 


r; 


i 


CONTROL  NUMBER:  Enter  a unique  control  number  in 
accordance  with  local  configuration  management  procedures. 

DATE:  Enter  the  date  the  DBCR  is  submitted. 

TO:  Enter  the  organization  responsible  for 

accomplishing  the  requested  change. 

FROM:  Enter  initiating  organization  and  the  name 

and  phone  number  of  the  individual  in  the  organization  ini- 
tiating the  DBCR  who  is  most  familiar  with  the  request. 

DATA  BASE  IDENTIFICATION:  Clearly  identify  the  data 
base  to  be  changed  through  implementation  of  the  request. 

MODULE  (S)  AFFECTED:  Identify  all  modules  affected 
to  insure  all  necessary  action  is  taken. 

FORM  OF  INPUT:  Check  one  listed.  If  "other," 
describe  in  REMARKS  block. 

REASON  FOR  CHANGE:  Self-explanatory. 

REMARKS:  Describe  form  of  input  if  "other"  is 

checked  above;  any  additional  information  which  could  assist 
in  accomplishing  the  change. 

CHANGE  IMPLEMENTED:  This  block  is  to  be  completed 
by  the  appropriate  review  official  when  change  is  accomplished. 


?7 


DATA  B^SE  CHANGE  REQUEST 


Control  Numbe 

TO:  Date  of  Reques 


FROM: 

Organization  

Point  of  Contact  Phone 


DATA  BASE  IDENTIFICATION 


MODULE (S)  AFFECTED  

FORM  OF  INPUT  Q D/B  Input  Requirements  Form 

Q Data  Cards  and  Listings 

Q 80-column  coding  sheets 

Q D/B  Tape  Containing  Data  Block, 
Reel  Number  

P Other  (Specify  Below) 

REASON  FOR  CHANGE 


REMARKS 


CHANGE  IMPLEMENTATION 


{J  APPROVED  SIGNATURE  DATE 


d.  BASELINE  CHANGE  REQUEST  (BCR) 


PROJECT  TITLE:  Enter  title  of  project. 

CHANGE  CONTROL  NUMBER:  Enter  a unique  control 
number  in  accordance  with  local  configuration  management 
procedures . 

DATE:  Enter  date  submitted. 

OFFICE  SYMBOL:  Enter  office  symbol  of  originator. 

TELEPHONE  NUMBER:  Enter  telephone  number  of  change 
request  originator. 

CHANGE  CLASS:  Self-explanatory. 

BASELINE  AND  BASELINE  DOCUMENTS  REQUIRING  CHANGE; 
Check  the  baseline (s)  or  document (s)  affected  by  the  change 
request. 

DESCRIPTION  OF  CHANGE:  A description  of  the 
requested  change  will  be  sufficient  in  detail  to  allow  for 
determination  of  project  impact.  Other  pertinent  facts  may 
be  attached. 

JUSTIFICATION:  This  will  contain  information 

sufficient  to  support  the  requested  change.  Generalities 
Ti.aSt  be  avoided. 

OBJECTIVES/BENEFITS:  Explain  what  should  be  achieved 

by  this  change.  Quantify  the  results  of  this  change  in 
terms  of  money,  time,  equipment,  etc. 

PROJECT  IMPACT;  Enter  such  information  as  increased 


costs,  schedule  delays,  product  changes,  etc. 


COST  ESTIMATES  CHANGE;  Provide  estimates  in  terms 


I 


(- 


! 

f 
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of  manhours  the  change  will  require.  Only  those  costs 
incurred  by  the  change  will  be  shown  and  not  total  develop- 
ment costs.  However,  the  total  costs  prior  to  this  change 
may  be  shown  for  comparison  purposes. 

DEVELOPER  SIGNATURE:  The  appropriate  approval/ 
disapproval  block  will  be  checked,  developer  (usually  proj- 
ect officer)  will  sign,  insert  his  office  symbol  and  date  of 
signature. 

USER  SIGNATURE:  The  appropriate  approval/disapproval 
block  will  be  checked,  user  will  sign,  insert  office  symbol 
and  date  of  signature.  It  is  not  necessary  to  obtain  user 
signature  for  Class  II  changes. 

MANAGEMENT  REVIEW:  Enter  the  review  that  evaluated 
this  change  (CCB  or  SDR) . The  approval  official  enters 
signature,  checks  either  the  approved  of  disapproved  block, 
and  enters  the  date  of  approval  or  disapproval. 


BASELlfiE  CIIAri'iE  RCQL'; 


PROJECT  IMPACT  STATEHERT 


COST  ESTIWTES 


FUNCTIONAL  ANALYST  MANHOURS 

DATA  SYSTEM  ANALYST  MANHOURS 
PROGRAMMER  MANHOURS 
OTHER  manhours 
total  MAmhOURS 
CONPUTER  SUPPORT  HOURS 
OTHER  COSTS 


OEVELOPER  SIGNATURE 


I — I APPROVED  C3  disapproved 


OFFICE  SYMSOL 


USER  SIGNATURE 


I — I APPROVED  LI3  DISAPPROVED 


OFFICE  SYMBOL 


WNAGEMENT  REVIEW 


1 — I APPRCVEO 


SIGNATURE 


CD  OISAFPROYED 
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